2024-09-16T02:51:03.600 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T02:51:03.620 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T02:51:03.941 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823 branch: main description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905823' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request mon_bind_msgr2: false sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP5pH6b1OqOYcSULw3nxTnOslOOs/XDZOgoqzzq6yKSlpb1pXa/bZ43/JPENvQ9V4I/dIR1pfVmYEMQ1vUke5G0= smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC7ukruT5mdjQVQQe3hkdNsgSc7XyHVGPdggN4PfWH53wxvE+xIiEcKKO8H8wvhO/yoCxz1U3Iusi9WsfFvUkiQ= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T02:51:03.941 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-16T02:51:03.942 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-16T02:51:03.942 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T02:51:03.944 INFO:teuthology.task.internal:Checking packages... 2024-09-16T02:51:03.975 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-16T02:51:03.975 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T02:51:03.975 INFO:teuthology.packaging:ref: None 2024-09-16T02:51:03.975 INFO:teuthology.packaging:tag: None 2024-09-16T02:51:03.976 INFO:teuthology.packaging:branch: main 2024-09-16T02:51:03.976 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:51:03.976 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-09-16T02:51:04.185 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4937-g74984932-1jammy 2024-09-16T02:51:04.187 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T02:51:04.194 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T02:51:04.194 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T02:51:04.241 INFO:teuthology.task.internal:Saving configuration 2024-09-16T02:51:04.256 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T02:51:04.280 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T02:51:04.304 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 02:45:24.478733', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMeVzCC9ikgKxwIKSG+nW3+WkdkGuONAQxWwMWaIKY+j'} 2024-09-16T02:51:04.327 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 02:45:24.480481', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIABqRbvOf2e3cQHllIQnFGpCxL8VSbcWuE9e1qjrbx93'} 2024-09-16T02:51:04.327 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T02:51:04.333 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T02:51:04.334 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T02:51:04.334 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T02:51:04.430 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4fe194ee60>, signals=[15]) 2024-09-16T02:51:04.430 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T02:51:04.437 INFO:teuthology.task.internal:Opening connections... 2024-09-16T02:51:04.437 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-09-16T02:51:04.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:51:04.523 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T02:51:04.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:51:04.612 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T02:51:04.620 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-09-16T02:51:04.625 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-09-16T02:51:04.626 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-09-16T02:51:04.677 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T02:51:04.677 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="Ubuntu" 2024-09-16T02:51:04.677 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="22.04" 2024-09-16T02:51:04.677 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T02:51:04.677 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_CODENAME=jammy 2024-09-16T02:51:04.677 INFO:teuthology.orchestra.run.smithi045.stdout:ID=ubuntu 2024-09-16T02:51:04.678 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE=debian 2024-09-16T02:51:04.678 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T02:51:04.678 INFO:teuthology.orchestra.run.smithi045.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T02:51:04.678 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T02:51:04.678 INFO:teuthology.orchestra.run.smithi045.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T02:51:04.678 INFO:teuthology.orchestra.run.smithi045.stdout:UBUNTU_CODENAME=jammy 2024-09-16T02:51:04.678 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-09-16T02:51:04.706 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-09-16T02:51:04.712 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-09-16T02:51:04.712 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-09-16T02:51:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T02:51:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="Ubuntu" 2024-09-16T02:51:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="22.04" 2024-09-16T02:51:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T02:51:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_CODENAME=jammy 2024-09-16T02:51:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:ID=ubuntu 2024-09-16T02:51:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE=debian 2024-09-16T02:51:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T02:51:04.761 INFO:teuthology.orchestra.run.smithi046.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T02:51:04.761 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T02:51:04.761 INFO:teuthology.orchestra.run.smithi046.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T02:51:04.761 INFO:teuthology.orchestra.run.smithi046.stdout:UBUNTU_CODENAME=jammy 2024-09-16T02:51:04.761 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-09-16T02:51:04.789 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T02:51:04.799 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T02:51:04.805 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T02:51:04.805 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-09-16T02:51:04.807 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-09-16T02:51:04.812 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T02:51:04.818 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T02:51:04.819 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-09-16T02:51:04.854 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-09-16T02:51:04.861 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T02:51:04.915 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T02:51:04.929 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T02:51:04.929 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T02:51:04.929 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T02:51:04.930 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T02:51:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:51:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:51:04.936 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-09-16T02:51:04.936 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-09-16T02:51:04.982 INFO:teuthology.orchestra.run.smithi045.stdout:5.15.0-56-generic 2024-09-16T02:51:04.982 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.15.0-56-generic 2024-09-16T02:51:04.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get clean 2024-09-16T02:51:04.984 INFO:teuthology.orchestra.run.smithi046.stdout:5.15.0-56-generic 2024-09-16T02:51:04.985 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.15.0-56-generic 2024-09-16T02:51:04.985 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get clean 2024-09-16T02:51:05.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2024-09-16T02:51:05.108 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2024-09-16T02:51:05.240 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:51:05.258 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T02:51:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:51:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T02:51:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T02:51:05.324 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T02:51:05.324 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T02:51:05.357 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T02:51:05.468 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T02:51:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T02:51:05.567 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T02:51:05.573 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T02:51:05.577 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T02:51:05.577 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T02:51:05.587 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T02:51:05.597 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T02:51:05.597 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T02:51:05.601 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T02:51:05.602 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T02:51:05.604 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T02:51:05.609 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T02:51:05.609 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T02:51:05.611 INFO:teuthology.orchestra.run.smithi045.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T02:51:05.611 INFO:teuthology.orchestra.run.smithi045.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T02:51:05.619 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T02:51:05.620 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T02:51:05.621 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T02:51:05.621 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T02:51:05.629 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T02:51:05.629 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T02:51:05.631 INFO:teuthology.orchestra.run.smithi045.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T02:51:05.632 INFO:teuthology.orchestra.run.smithi045.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T02:51:05.635 INFO:teuthology.orchestra.run.smithi045.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T02:51:05.662 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T02:51:05.662 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T02:51:05.663 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T02:51:05.663 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T02:51:05.663 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T02:51:05.663 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T02:51:05.663 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T02:51:05.663 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T02:51:05.664 INFO:teuthology.orchestra.run.smithi045.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T02:51:05.664 INFO:teuthology.orchestra.run.smithi045.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T02:51:05.672 INFO:teuthology.orchestra.run.smithi045.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T02:51:05.678 INFO:teuthology.orchestra.run.smithi045.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T02:51:05.680 INFO:teuthology.orchestra.run.smithi045.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T02:51:05.680 INFO:teuthology.orchestra.run.smithi045.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T02:51:05.681 INFO:teuthology.orchestra.run.smithi045.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T02:51:05.681 INFO:teuthology.orchestra.run.smithi045.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T02:51:05.682 INFO:teuthology.orchestra.run.smithi045.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T02:51:05.702 INFO:teuthology.orchestra.run.smithi045.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T02:51:05.706 INFO:teuthology.orchestra.run.smithi045.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T02:51:05.706 INFO:teuthology.orchestra.run.smithi045.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T02:51:05.707 INFO:teuthology.orchestra.run.smithi045.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T02:51:05.728 INFO:teuthology.orchestra.run.smithi045.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T02:51:05.732 INFO:teuthology.orchestra.run.smithi045.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T02:51:05.733 INFO:teuthology.orchestra.run.smithi045.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T02:51:05.739 INFO:teuthology.orchestra.run.smithi045.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T02:51:05.748 INFO:teuthology.orchestra.run.smithi045.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T02:51:05.750 INFO:teuthology.orchestra.run.smithi045.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T02:51:05.751 INFO:teuthology.orchestra.run.smithi045.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T02:51:05.752 INFO:teuthology.orchestra.run.smithi045.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T02:51:05.752 INFO:teuthology.orchestra.run.smithi045.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T02:51:05.769 INFO:teuthology.orchestra.run.smithi045.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T02:51:05.770 INFO:teuthology.orchestra.run.smithi045.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T02:51:05.770 INFO:teuthology.orchestra.run.smithi045.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T02:51:05.775 INFO:teuthology.orchestra.run.smithi045.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T02:51:05.776 INFO:teuthology.orchestra.run.smithi045.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T02:51:05.776 INFO:teuthology.orchestra.run.smithi045.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T02:51:05.776 INFO:teuthology.orchestra.run.smithi045.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T02:51:05.793 INFO:teuthology.orchestra.run.smithi045.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T02:51:05.793 INFO:teuthology.orchestra.run.smithi045.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T02:51:05.819 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T02:51:06.240 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T02:51:06.256 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T02:51:06.265 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T02:51:06.265 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T02:51:06.362 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T02:51:06.363 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T02:51:06.374 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T02:51:06.374 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T02:51:06.398 INFO:teuthology.orchestra.run.smithi046.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T02:51:06.420 INFO:teuthology.orchestra.run.smithi046.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T02:51:06.422 INFO:teuthology.orchestra.run.smithi046.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T02:51:06.425 INFO:teuthology.orchestra.run.smithi046.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T02:51:06.426 INFO:teuthology.orchestra.run.smithi046.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T02:51:06.501 INFO:teuthology.orchestra.run.smithi046.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T02:51:06.502 INFO:teuthology.orchestra.run.smithi046.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T02:51:12.304 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 16.0 MB in 2s (7,222 kB/s) 2024-09-16T02:51:12.377 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 16.0 MB in 2s (7,083 kB/s) 2024-09-16T02:51:13.780 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-16T02:51:13.799 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:51:13.854 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-16T02:51:13.865 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-09-16T02:51:13.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:51:13.946 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-09-16T02:51:14.051 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2024-09-16T02:51:14.051 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2024-09-16T02:51:14.159 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2024-09-16T02:51:14.160 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2024-09-16T02:51:14.229 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:51:14.229 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:51:14.229 INFO:teuthology.orchestra.run.smithi046.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:51:14.229 INFO:teuthology.orchestra.run.smithi046.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:51:14.230 INFO:teuthology.orchestra.run.smithi046.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T02:51:14.230 INFO:teuthology.orchestra.run.smithi046.stdout: python2.7-minimal 2024-09-16T02:51:14.230 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:51:14.231 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2024-09-16T02:51:14.231 INFO:teuthology.orchestra.run.smithi046.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T02:51:14.231 INFO:teuthology.orchestra.run.smithi046.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T02:51:14.231 INFO:teuthology.orchestra.run.smithi046.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T02:51:14.232 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2024-09-16T02:51:14.233 INFO:teuthology.orchestra.run.smithi046.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T02:51:14.233 INFO:teuthology.orchestra.run.smithi046.stdout:Recommended packages: 2024-09-16T02:51:14.233 INFO:teuthology.orchestra.run.smithi046.stdout: thermald 2024-09-16T02:51:14.284 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2024-09-16T02:51:14.284 INFO:teuthology.orchestra.run.smithi046.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T02:51:14.285 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T02:51:14.285 INFO:teuthology.orchestra.run.smithi046.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T02:51:14.286 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages will be upgraded: 2024-09-16T02:51:14.287 INFO:teuthology.orchestra.run.smithi046.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T02:51:14.338 INFO:teuthology.orchestra.run.smithi046.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:51:14.338 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 113 MB of archives. 2024-09-16T02:51:14.338 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T02:51:14.338 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T02:51:14.356 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:51:14.356 INFO:teuthology.orchestra.run.smithi045.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:51:14.356 INFO:teuthology.orchestra.run.smithi045.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:51:14.356 INFO:teuthology.orchestra.run.smithi045.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:51:14.356 INFO:teuthology.orchestra.run.smithi045.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T02:51:14.357 INFO:teuthology.orchestra.run.smithi045.stdout: python2.7-minimal 2024-09-16T02:51:14.357 INFO:teuthology.orchestra.run.smithi045.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:51:14.358 INFO:teuthology.orchestra.run.smithi045.stdout:The following additional packages will be installed: 2024-09-16T02:51:14.358 INFO:teuthology.orchestra.run.smithi045.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T02:51:14.358 INFO:teuthology.orchestra.run.smithi045.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T02:51:14.359 INFO:teuthology.orchestra.run.smithi045.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T02:51:14.360 INFO:teuthology.orchestra.run.smithi045.stdout:Suggested packages: 2024-09-16T02:51:14.361 INFO:teuthology.orchestra.run.smithi045.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T02:51:14.361 INFO:teuthology.orchestra.run.smithi045.stdout:Recommended packages: 2024-09-16T02:51:14.361 INFO:teuthology.orchestra.run.smithi045.stdout: thermald 2024-09-16T02:51:14.414 INFO:teuthology.orchestra.run.smithi045.stdout:The following NEW packages will be installed: 2024-09-16T02:51:14.414 INFO:teuthology.orchestra.run.smithi045.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T02:51:14.414 INFO:teuthology.orchestra.run.smithi045.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T02:51:14.415 INFO:teuthology.orchestra.run.smithi045.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T02:51:14.416 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages will be upgraded: 2024-09-16T02:51:14.417 INFO:teuthology.orchestra.run.smithi045.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T02:51:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:51:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:Need to get 113 MB of archives. 2024-09-16T02:51:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T02:51:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T02:51:14.757 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T02:51:14.850 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T02:51:15.656 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T02:51:15.656 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T02:51:15.656 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T02:51:15.786 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T02:51:15.808 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T02:51:15.925 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T02:51:16.221 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 113 MB in 2s (74.8 MB/s) 2024-09-16T02:51:16.276 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T02:51:16.377 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T02:51:18.381 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T02:51:18.385 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:51:18.493 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T02:51:18.493 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T02:51:18.493 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T02:51:18.524 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:18.918 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T02:51:18.981 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T02:51:19.397 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 113 MB in 5s (24.9 MB/s) 2024-09-16T02:51:19.523 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T02:51:20.835 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T02:51:20.856 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:51:20.945 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:21.397 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T02:51:21.418 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:51:21.498 INFO:teuthology.orchestra.run.smithi045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T02:51:21.503 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:51:21.563 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:21.587 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:24.265 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T02:51:24.286 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:51:24.368 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:24.818 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T02:51:24.839 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:51:24.902 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:28.202 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:51:28.354 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:51:28.686 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:51:28.833 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:51:29.143 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T02:51:29.154 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T02:51:29.185 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:51:31.616 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:51:31.724 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:51:32.032 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:51:32.127 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:51:32.498 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T02:51:32.521 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T02:51:32.564 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:51:35.812 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T02:51:35.849 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:51:35.887 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:38.586 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T02:51:38.623 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:51:38.660 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:38.755 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:51:38.912 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:51:39.256 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:51:39.385 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:39.536 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T02:51:39.655 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:41.065 INFO:teuthology.orchestra.run.smithi046.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:51:41.066 INFO:teuthology.orchestra.run.smithi046.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T02:51:41.066 INFO:teuthology.orchestra.run.smithi046.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:51:41.066 INFO:teuthology.orchestra.run.smithi046.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T02:51:41.859 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:42.290 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:51:42.424 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:51:42.816 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:51:42.913 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:43.030 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T02:51:43.158 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:43.309 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:44.507 INFO:teuthology.orchestra.run.smithi045.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:51:44.508 INFO:teuthology.orchestra.run.smithi045.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T02:51:44.508 INFO:teuthology.orchestra.run.smithi045.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:51:44.508 INFO:teuthology.orchestra.run.smithi045.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T02:51:44.679 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:44.769 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T02:51:44.936 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T02:51:45.079 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:45.165 INFO:teuthology.orchestra.run.smithi046.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T02:51:45.165 INFO:teuthology.orchestra.run.smithi046.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:51:46.172 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:47.973 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T02:51:48.099 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T02:51:48.225 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:51:48.287 INFO:teuthology.orchestra.run.smithi045.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T02:51:48.287 INFO:teuthology.orchestra.run.smithi045.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:51:59.765 INFO:teuthology.orchestra.run.smithi046.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T02:51:59.765 INFO:teuthology.orchestra.run.smithi046.stdout:Sourcing file `/etc/default/grub' 2024-09-16T02:51:59.790 INFO:teuthology.orchestra.run.smithi046.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:51:59.808 INFO:teuthology.orchestra.run.smithi046.stdout:Generating grub configuration file ... 2024-09-16T02:52:00.193 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:52:00.211 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:52:00.634 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:52:00.640 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:52:00.702 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:52:00.707 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:52:00.976 INFO:teuthology.orchestra.run.smithi046.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:52:00.976 INFO:teuthology.orchestra.run.smithi046.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:52:00.976 INFO:teuthology.orchestra.run.smithi046.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:52:01.022 INFO:teuthology.orchestra.run.smithi046.stdout:done 2024-09-16T02:52:01.604 DEBUG:teuthology.orchestra.run.smithi046:> dpkg -s linux-image-generic 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Package: linux-image-generic 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Status: install ok installed 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Priority: optional 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Section: kernel 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Installed-Size: 21 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: amd64 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Source: linux-meta 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Version: 5.15.0.121.121 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:52:01.630 INFO:teuthology.orchestra.run.smithi046.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:52:01.631 INFO:teuthology.orchestra.run.smithi046.stdout:Recommends: thermald 2024-09-16T02:52:01.631 INFO:teuthology.orchestra.run.smithi046.stdout:Description: Generic Linux kernel image 2024-09-16T02:52:01.631 INFO:teuthology.orchestra.run.smithi046.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:52:01.631 INFO:teuthology.orchestra.run.smithi046.stdout: available. 2024-09-16T02:52:01.631 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T02:52:01.631 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T02:52:01.631 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T02:52:01.631 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T02:52:01.632 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T02:52:01.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get clean 2024-09-16T02:52:01.805 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2024-09-16T02:52:02.059 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:52:02.059 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:52:02.130 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:52:02.211 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:52:02.586 INFO:teuthology.orchestra.run.smithi045.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T02:52:02.586 INFO:teuthology.orchestra.run.smithi045.stdout:Sourcing file `/etc/default/grub' 2024-09-16T02:52:02.591 INFO:teuthology.orchestra.run.smithi045.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:52:02.604 INFO:teuthology.orchestra.run.smithi045.stdout:Generating grub configuration file ... 2024-09-16T02:52:02.856 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:52:02.890 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:52:03.174 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:52:03.180 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:52:03.249 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:52:03.255 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:52:03.407 INFO:teuthology.orchestra.run.smithi045.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:52:03.407 INFO:teuthology.orchestra.run.smithi045.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:52:03.408 INFO:teuthology.orchestra.run.smithi045.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:52:03.450 INFO:teuthology.orchestra.run.smithi045.stdout:done 2024-09-16T02:52:03.540 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-16T02:52:03.559 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:52:03.626 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-16T02:52:03.844 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2024-09-16T02:52:03.845 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2024-09-16T02:52:03.942 DEBUG:teuthology.orchestra.run.smithi045:> dpkg -s linux-image-generic 2024-09-16T02:52:03.967 INFO:teuthology.orchestra.run.smithi045.stdout:Package: linux-image-generic 2024-09-16T02:52:03.967 INFO:teuthology.orchestra.run.smithi045.stdout:Status: install ok installed 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Priority: optional 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Section: kernel 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Installed-Size: 21 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: amd64 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Source: linux-meta 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Version: 5.15.0.121.121 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Recommends: thermald 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout:Description: Generic Linux kernel image 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:52:03.968 INFO:teuthology.orchestra.run.smithi045.stdout: available. 2024-09-16T02:52:03.969 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T02:52:03.969 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T02:52:03.969 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T02:52:03.969 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T02:52:03.969 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T02:52:03.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get clean 2024-09-16T02:52:04.043 INFO:teuthology.orchestra.run.smithi046.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T02:52:04.043 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:52:04.043 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:52:04.044 INFO:teuthology.orchestra.run.smithi046.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:52:04.044 INFO:teuthology.orchestra.run.smithi046.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:52:04.044 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:52:04.044 INFO:teuthology.orchestra.run.smithi046.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:52:04.044 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:52:04.092 INFO:teuthology.orchestra.run.smithi046.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:52:04.094 DEBUG:teuthology.orchestra.run.smithi046:> dpkg -s linux-image-generic 2024-09-16T02:52:04.107 INFO:teuthology.orchestra.run.smithi046.stdout:Package: linux-image-generic 2024-09-16T02:52:04.107 INFO:teuthology.orchestra.run.smithi046.stdout:Status: install ok installed 2024-09-16T02:52:04.107 INFO:teuthology.orchestra.run.smithi046.stdout:Priority: optional 2024-09-16T02:52:04.107 INFO:teuthology.orchestra.run.smithi046.stdout:Section: kernel 2024-09-16T02:52:04.107 INFO:teuthology.orchestra.run.smithi046.stdout:Installed-Size: 21 2024-09-16T02:52:04.107 INFO:teuthology.orchestra.run.smithi046.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:52:04.108 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: amd64 2024-09-16T02:52:04.108 INFO:teuthology.orchestra.run.smithi046.stdout:Source: linux-meta 2024-09-16T02:52:04.108 INFO:teuthology.orchestra.run.smithi046.stdout:Version: 5.15.0.121.121 2024-09-16T02:52:04.108 INFO:teuthology.orchestra.run.smithi046.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:52:04.108 INFO:teuthology.orchestra.run.smithi046.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:52:04.108 INFO:teuthology.orchestra.run.smithi046.stdout:Recommends: thermald 2024-09-16T02:52:04.108 INFO:teuthology.orchestra.run.smithi046.stdout:Description: Generic Linux kernel image 2024-09-16T02:52:04.108 INFO:teuthology.orchestra.run.smithi046.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:52:04.108 INFO:teuthology.orchestra.run.smithi046.stdout: available. 2024-09-16T02:52:04.108 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-09-16T02:52:04.156 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.1JurdUnrPg 2024-09-16T02:52:04.156 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub/grub.cfg /tmp/tmp.1JurdUnrPg 2024-09-16T02:52:04.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2024-09-16T02:52:04.213 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.1JurdUnrPg 2024-09-16T02:52:04.301 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:52:04.343 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.1JurdUnrPg is 10KB 2024-09-16T02:52:04.356 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.1JurdUnrPg 2024-09-16T02:52:04.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T02:52:04.417 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T02:52:04.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T02:52:04.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T02:52:04.422 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:52:04.483 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T02:52:04.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo update-grub 2024-09-16T02:52:04.503 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:52:04.584 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:52:05.749 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-09-16T02:52:05.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:52:05.835 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-09-16T02:52:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub' 2024-09-16T02:52:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:52:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-09-16T02:52:06.055 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2024-09-16T02:52:06.055 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2024-09-16T02:52:06.252 INFO:teuthology.orchestra.run.smithi045.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T02:52:06.252 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:52:06.253 INFO:teuthology.orchestra.run.smithi045.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:52:06.253 INFO:teuthology.orchestra.run.smithi045.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:52:06.253 INFO:teuthology.orchestra.run.smithi045.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:52:06.253 INFO:teuthology.orchestra.run.smithi045.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:52:06.253 INFO:teuthology.orchestra.run.smithi045.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:52:06.253 INFO:teuthology.orchestra.run.smithi045.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:52:06.288 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:52:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:52:06.298 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:52:06.300 DEBUG:teuthology.orchestra.run.smithi045:> dpkg -s linux-image-generic 2024-09-16T02:52:06.316 INFO:teuthology.orchestra.run.smithi045.stdout:Package: linux-image-generic 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Status: install ok installed 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Priority: optional 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Section: kernel 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Installed-Size: 21 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: amd64 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Source: linux-meta 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Version: 5.15.0.121.121 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:52:06.317 INFO:teuthology.orchestra.run.smithi045.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:52:06.318 INFO:teuthology.orchestra.run.smithi045.stdout:Recommends: thermald 2024-09-16T02:52:06.318 INFO:teuthology.orchestra.run.smithi045.stdout:Description: Generic Linux kernel image 2024-09-16T02:52:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:52:06.318 INFO:teuthology.orchestra.run.smithi045.stdout: available. 2024-09-16T02:52:06.318 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-09-16T02:52:06.365 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.1EkbHL5hcP 2024-09-16T02:52:06.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub/grub.cfg /tmp/tmp.1EkbHL5hcP 2024-09-16T02:52:06.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.1EkbHL5hcP 2024-09-16T02:52:06.542 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.1EkbHL5hcP is 10KB 2024-09-16T02:52:06.557 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.1EkbHL5hcP 2024-09-16T02:52:06.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T02:52:06.616 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T02:52:06.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T02:52:06.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T02:52:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:52:06.653 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:52:06.695 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T02:52:06.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo update-grub 2024-09-16T02:52:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:52:06.707 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:52:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:52:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:52:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:52:06.862 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-09-16T02:52:06.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-09-16T02:52:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:Sourcing file `/etc/default/grub' 2024-09-16T02:52:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:52:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-09-16T02:52:08.131 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:52:08.137 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:52:08.380 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:52:08.386 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:52:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:52:08.438 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:52:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:52:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:52:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:52:08.603 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-09-16T02:52:08.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-09-16T02:52:36.893 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T02:52:36.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T02:52:36.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:52:38.615 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T02:52:38.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-16T02:52:38.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:52:55.229 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-09-16T02:52:57.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-09-16T02:53:04.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T02:53:04.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:53:06.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-16T02:53:06.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:53:07.293 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-09-16T02:53:19.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T02:53:19.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:54:06.074 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T02:54:18.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-16T02:54:18.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:54:18.383 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-09-16T02:54:18.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-16T02:54:18.969 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T02:54:18.970 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-09-16T02:54:19.018 INFO:teuthology.orchestra.run.smithi045.stdout:5.15.0-121-generic 2024-09-16T02:54:19.018 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T02:54:19.018 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T02:54:19.018 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T02:54:19.307 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T02:54:20.019 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T02:54:20.020 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T02:54:20.084 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-09-16T02:54:20.116 DEBUG:teuthology.parallel:result is None 2024-09-16T02:54:34.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T02:54:34.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:54:34.527 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-09-16T02:54:35.192 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T02:54:35.192 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-16T02:54:35.192 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-09-16T02:54:35.238 INFO:teuthology.orchestra.run.smithi046.stdout:5.15.0-121-generic 2024-09-16T02:54:35.238 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T02:54:35.238 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T02:54:35.239 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T02:54:36.239 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T02:54:36.239 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T02:54:36.307 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-09-16T02:54:36.331 DEBUG:teuthology.parallel:result is None 2024-09-16T02:54:36.331 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T02:54:36.339 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T02:54:36.339 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T02:54:36.342 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T02:54:36.347 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T02:54:36.354 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T02:54:36.362 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T02:54:36.363 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T02:54:36.391 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T02:54:36.413 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T02:54:36.421 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T02:54:36.421 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T02:54:36.451 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T02:54:36.471 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:54:36.472 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:54:36.480 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:54:36.482 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:54:36.484 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T02:54:36.494 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T02:54:36.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T02:54:36.527 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T02:54:36.541 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T02:54:36.554 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T02:54:36.554 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T02:54:36.583 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T02:54:36.590 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T02:54:36.634 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T02:54:36.682 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T02:54:36.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T02:54:36.747 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T02:54:36.753 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T02:54:36.802 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T02:54:36.802 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T02:54:36.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-09-16T02:54:36.863 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-09-16T02:54:36.939 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T02:54:36.950 INFO:teuthology.task.internal:Starting timer... 2024-09-16T02:54:36.950 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T02:54:36.995 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T02:54:37.008 DEBUG:teuthology.task.selinux:Excluding smithi045: OS 'ubuntu' does not support SELinux 2024-09-16T02:54:37.008 DEBUG:teuthology.task.selinux:Excluding smithi046: OS 'ubuntu' does not support SELinux 2024-09-16T02:54:37.009 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T02:54:37.009 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T02:54:37.009 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T02:54:37.009 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T02:54:37.021 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T02:54:37.024 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T02:54:37.137 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T02:54:37.148 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T02:54:37.149 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi046.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T03:02:15.880 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi046.front.sepia.ceph.com')] 2024-09-16T03:02:15.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-16T03:02:15.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:02:15.966 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-09-16T03:02:16.045 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-16T03:02:16.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T03:02:16.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:02:16.133 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-09-16T03:02:16.213 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T03:02:16.213 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T03:02:16.497 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T03:02:16.497 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:02:16.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:02:16.501 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:02:16.501 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:02:16.531 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T03:02:16.531 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: Command line: ntpd -gq 2024-09-16T03:02:16.531 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: ---------------------------------------------------- 2024-09-16T03:02:16.531 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T03:02:16.531 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T03:02:16.531 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: corporation. Support and training for ntp-4 are 2024-09-16T03:02:16.531 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: available at https://www.nwtime.org/support 2024-09-16T03:02:16.531 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: ---------------------------------------------------- 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: Command line: ntpd -gq 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: ---------------------------------------------------- 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: corporation. Support and training for ntp-4 are 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: available at https://www.nwtime.org/support 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: ---------------------------------------------------- 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: proto: precision = 0.046 usec (-24) 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: basedate set to 2022-02-04 2024-09-16T03:02:16.533 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: gps base set to 2022-02-06 (week 2196) 2024-09-16T03:02:16.534 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: proto: precision = 0.067 usec (-24) 2024-09-16T03:02:16.534 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: basedate set to 2022-02-04 2024-09-16T03:02:16.534 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: gps base set to 2022-02-06 (week 2196) 2024-09-16T03:02:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:02:16.534 INFO:teuthology.orchestra.run.smithi045.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:02:16.535 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:02:16.535 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:02:16.535 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:02:16.535 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:02:16.535 INFO:teuthology.orchestra.run.smithi046.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:02:16.535 INFO:teuthology.orchestra.run.smithi045.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:02:16.537 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T03:02:16.537 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T03:02:16.537 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T03:02:16.537 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: Listen normally on 3 ens1f0 172.21.15.46:123 2024-09-16T03:02:16.537 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: Listen normally on 4 lo [::1]:123 2024-09-16T03:02:16.537 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15f8%4]:123 2024-09-16T03:02:16.537 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:16 ntpd[18368]: Listening on routing socket on fd #22 for interface updates 2024-09-16T03:02:16.538 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T03:02:16.538 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T03:02:16.538 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T03:02:16.538 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: Listen normally on 3 ens1f0 172.21.15.45:123 2024-09-16T03:02:16.538 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: Listen normally on 4 lo [::1]:123 2024-09-16T03:02:16.538 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:12be%4]:123 2024-09-16T03:02:16.538 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:16 ntpd[18365]: Listening on routing socket on fd #22 for interface updates 2024-09-16T03:02:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:17 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:17 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:17 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:17 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:17 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:17 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:17.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:17.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:17.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:17.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:17.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:17.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:17.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:17.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:17.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:17.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:17.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:17.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:17 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:18.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:18 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:18 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:18 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:18 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:18 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:18 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:18 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:19.531 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:19 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:19.531 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:19 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:19.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:19 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:19.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:19 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:19.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:19 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:19.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:19 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:19.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:19.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:19.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:19.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:19 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:20.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:20 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:20 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:20 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:20 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:20 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:20 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:20 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:21 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:21 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:21 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:21 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:21 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:21 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:21.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:21.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:21.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:21.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:21.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:21.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:21 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:22.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:22.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:22.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:22.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:22.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:22 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:22 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:22 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:22 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:22 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:22 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:22 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:23.531 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:23 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:23.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:23 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:23.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:23 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:23.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:23 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:23.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:23 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:23.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:23 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:23.532 INFO:teuthology.orchestra.run.smithi045.stdout:16 Sep 03:02:23 ntpd[18365]: ntpd: time slew -0.000496 s 2024-09-16T03:02:23.532 INFO:teuthology.orchestra.run.smithi045.stdout:ntpd: time slew -0.000496s 2024-09-16T03:02:23.532 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:23 ntpd[18365]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T03:02:23.533 INFO:teuthology.orchestra.run.smithi045.stderr:16 Sep 03:02:23 ntpd[18365]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T03:02:23.533 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:23.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:23.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:23.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:23.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:02:23.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:02:23.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:02:23.534 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:02:23.534 INFO:teuthology.orchestra.run.smithi046.stdout:16 Sep 03:02:23 ntpd[18368]: ntpd: time slew -0.000253 s 2024-09-16T03:02:23.534 INFO:teuthology.orchestra.run.smithi046.stdout:ntpd: time slew -0.000253s 2024-09-16T03:02:23.535 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T03:02:23.535 INFO:teuthology.orchestra.run.smithi046.stderr:16 Sep 03:02:23 ntpd[18368]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T03:02:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:02:23.580 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================== 2024-09-16T03:02:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:02:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:02:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:02:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:02:23.584 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:02:23.584 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2024-09-16T03:02:23.585 INFO:teuthology.orchestra.run.smithi046.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:02:23.585 INFO:teuthology.orchestra.run.smithi046.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:02:23.585 INFO:teuthology.orchestra.run.smithi046.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:02:23.585 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:02:23.586 INFO:teuthology.run_tasks:Running task install... 2024-09-16T03:02:23.594 DEBUG:teuthology.task.install:project ceph 2024-09-16T03:02:23.594 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-16T03:02:23.594 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-16T03:02:23.594 INFO:teuthology.task.install:Using flavor: default 2024-09-16T03:02:23.600 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T03:02:23.600 INFO:teuthology.task.install:extra packages: [] 2024-09-16T03:02:23.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-key list | grep Ceph 2024-09-16T03:02:23.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-key list | grep Ceph 2024-09-16T03:02:23.687 INFO:teuthology.orchestra.run.smithi045.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T03:02:23.688 INFO:teuthology.orchestra.run.smithi046.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T03:02:23.716 INFO:teuthology.orchestra.run.smithi045.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T03:02:23.716 INFO:teuthology.orchestra.run.smithi045.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T03:02:23.717 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T03:02:23.717 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T03:02:23.718 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T03:02:23.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:02:23.721 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T03:02:23.721 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:02:23.863 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-16T03:02:23.863 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T03:02:23.866 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-16T03:02:23.866 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T03:02:23.977 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:02:23.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T03:02:23.983 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:02:23.984 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T03:02:23.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2024-09-16T03:02:23.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2024-09-16T03:02:24.180 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:02:24.216 INFO:teuthology.orchestra.run.smithi045.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T03:02:24.225 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T03:02:24.236 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-16T03:02:24.246 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-16T03:02:24.259 INFO:teuthology.orchestra.run.smithi045.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T03:02:24.270 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T03:02:24.281 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:02:24.283 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-16T03:02:24.291 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-16T03:02:24.310 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-16T03:02:24.311 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:02:24.311 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:02:24.332 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-16T03:02:24.364 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:02:24.374 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:02:24.445 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:02:24.456 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:02:24.585 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 29.2 kB in 0s (71.1 kB/s) 2024-09-16T03:02:24.701 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 29.2 kB in 1s (57.3 kB/s) 2024-09-16T03:02:25.620 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-09-16T03:02:25.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T03:02:25.694 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-09-16T03:02:25.740 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-16T03:02:25.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T03:02:25.813 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-16T03:02:25.883 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2024-09-16T03:02:25.884 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2024-09-16T03:02:26.006 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2024-09-16T03:02:26.007 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2024-09-16T03:02:26.077 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:02:26.077 INFO:teuthology.orchestra.run.smithi045.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:02:26.077 INFO:teuthology.orchestra.run.smithi045.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:02:26.077 INFO:teuthology.orchestra.run.smithi045.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:02:26.078 INFO:teuthology.orchestra.run.smithi045.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:02:26.078 INFO:teuthology.orchestra.run.smithi045.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:02:26.078 INFO:teuthology.orchestra.run.smithi045.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:02:26.079 INFO:teuthology.orchestra.run.smithi045.stdout:The following additional packages will be installed: 2024-09-16T03:02:26.079 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T03:02:26.079 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T03:02:26.079 INFO:teuthology.orchestra.run.smithi045.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T03:02:26.079 INFO:teuthology.orchestra.run.smithi045.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T03:02:26.079 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-16T03:02:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-16T03:02:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T03:02:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T03:02:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-16T03:02:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-16T03:02:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-16T03:02:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-16T03:02:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-16T03:02:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet zip 2024-09-16T03:02:26.081 INFO:teuthology.orchestra.run.smithi045.stdout:Suggested packages: 2024-09-16T03:02:26.082 INFO:teuthology.orchestra.run.smithi045.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T03:02:26.082 INFO:teuthology.orchestra.run.smithi045.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T03:02:26.082 INFO:teuthology.orchestra.run.smithi045.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T03:02:26.082 INFO:teuthology.orchestra.run.smithi045.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T03:02:26.082 INFO:teuthology.orchestra.run.smithi045.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T03:02:26.082 INFO:teuthology.orchestra.run.smithi045.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T03:02:26.082 INFO:teuthology.orchestra.run.smithi045.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T03:02:26.082 INFO:teuthology.orchestra.run.smithi045.stdout:Recommended packages: 2024-09-16T03:02:26.082 INFO:teuthology.orchestra.run.smithi045.stdout: btrfs-tools 2024-09-16T03:02:26.189 INFO:teuthology.orchestra.run.smithi045.stdout:The following NEW packages will be installed: 2024-09-16T03:02:26.189 INFO:teuthology.orchestra.run.smithi045.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T03:02:26.189 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T03:02:26.189 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T03:02:26.189 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T03:02:26.189 INFO:teuthology.orchestra.run.smithi045.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-16T03:02:26.189 INFO:teuthology.orchestra.run.smithi045.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T03:02:26.191 INFO:teuthology.orchestra.run.smithi045.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T03:02:26.191 INFO:teuthology.orchestra.run.smithi045.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T03:02:26.191 INFO:teuthology.orchestra.run.smithi045.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T03:02:26.191 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T03:02:26.191 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T03:02:26.191 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T03:02:26.192 INFO:teuthology.orchestra.run.smithi045.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T03:02:26.199 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:02:26.199 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:02:26.200 INFO:teuthology.orchestra.run.smithi046.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:02:26.200 INFO:teuthology.orchestra.run.smithi046.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:02:26.200 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:02:26.200 INFO:teuthology.orchestra.run.smithi046.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:02:26.200 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:02:26.201 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2024-09-16T03:02:26.201 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T03:02:26.201 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T03:02:26.202 INFO:teuthology.orchestra.run.smithi046.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T03:02:26.202 INFO:teuthology.orchestra.run.smithi046.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T03:02:26.202 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-16T03:02:26.202 INFO:teuthology.orchestra.run.smithi046.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-16T03:02:26.203 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-16T03:02:26.204 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T03:02:26.204 INFO:teuthology.orchestra.run.smithi046.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T03:02:26.204 INFO:teuthology.orchestra.run.smithi046.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T03:02:26.204 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T03:02:26.204 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T03:02:26.204 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet zip 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout:Recommended packages: 2024-09-16T03:02:26.205 INFO:teuthology.orchestra.run.smithi046.stdout: btrfs-tools 2024-09-16T03:02:26.310 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2024-09-16T03:02:26.310 INFO:teuthology.orchestra.run.smithi046.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T03:02:26.311 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T03:02:26.311 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T03:02:26.311 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T03:02:26.311 INFO:teuthology.orchestra.run.smithi046.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-16T03:02:26.311 INFO:teuthology.orchestra.run.smithi046.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T03:02:26.312 INFO:teuthology.orchestra.run.smithi046.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T03:02:26.312 INFO:teuthology.orchestra.run.smithi046.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T03:02:26.312 INFO:teuthology.orchestra.run.smithi046.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T03:02:26.312 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T03:02:26.312 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T03:02:26.312 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:02:26.312 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T03:02:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T03:02:26.334 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T03:02:26.334 INFO:teuthology.orchestra.run.smithi045.stdout:Need to get 244 MB of archives. 2024-09-16T03:02:26.334 INFO:teuthology.orchestra.run.smithi045.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-16T03:02:26.334 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-16T03:02:26.378 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T03:02:26.460 INFO:teuthology.orchestra.run.smithi046.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T03:02:26.460 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 244 MB of archives. 2024-09-16T03:02:26.460 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-16T03:02:26.460 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-16T03:02:26.504 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T03:02:26.605 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-16T03:02:26.721 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-16T03:02:26.742 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-16T03:02:26.753 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-16T03:02:26.757 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-16T03:02:26.758 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-16T03:02:26.762 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-16T03:02:26.764 INFO:teuthology.orchestra.run.smithi045.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-16T03:02:26.776 INFO:teuthology.orchestra.run.smithi045.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-16T03:02:26.802 INFO:teuthology.orchestra.run.smithi045.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T03:02:26.815 INFO:teuthology.orchestra.run.smithi045.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T03:02:26.871 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-16T03:02:26.896 INFO:teuthology.orchestra.run.smithi045.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T03:02:26.897 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-16T03:02:26.917 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-16T03:02:26.918 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-16T03:02:26.922 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-16T03:02:26.924 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-16T03:02:26.927 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T03:02:26.940 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T03:02:26.944 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-16T03:02:26.995 INFO:teuthology.orchestra.run.smithi045.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-16T03:02:26.997 INFO:teuthology.orchestra.run.smithi045.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-16T03:02:27.014 INFO:teuthology.orchestra.run.smithi045.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-16T03:02:27.021 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T03:02:27.124 INFO:teuthology.orchestra.run.smithi045.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T03:02:27.136 INFO:teuthology.orchestra.run.smithi045.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T03:02:27.165 INFO:teuthology.orchestra.run.smithi045.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T03:02:27.173 INFO:teuthology.orchestra.run.smithi045.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-16T03:02:27.176 INFO:teuthology.orchestra.run.smithi045.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T03:02:27.177 INFO:teuthology.orchestra.run.smithi045.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T03:02:27.178 INFO:teuthology.orchestra.run.smithi045.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T03:02:27.196 INFO:teuthology.orchestra.run.smithi045.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T03:02:27.200 INFO:teuthology.orchestra.run.smithi045.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T03:02:27.206 INFO:teuthology.orchestra.run.smithi045.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T03:02:27.250 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T03:02:27.261 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T03:02:27.263 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-16T03:02:27.265 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-16T03:02:27.275 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-16T03:02:27.286 INFO:teuthology.orchestra.run.smithi045.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T03:02:27.287 INFO:teuthology.orchestra.run.smithi045.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T03:02:27.291 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T03:02:27.292 INFO:teuthology.orchestra.run.smithi045.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T03:02:27.296 INFO:teuthology.orchestra.run.smithi045.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T03:02:27.299 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-16T03:02:27.300 INFO:teuthology.orchestra.run.smithi045.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T03:02:27.301 INFO:teuthology.orchestra.run.smithi045.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T03:02:27.302 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T03:02:27.302 INFO:teuthology.orchestra.run.smithi045.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T03:02:27.303 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T03:02:27.304 INFO:teuthology.orchestra.run.smithi045.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T03:02:27.304 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T03:02:27.305 INFO:teuthology.orchestra.run.smithi045.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T03:02:27.321 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T03:02:27.328 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T03:02:27.332 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T03:02:27.369 INFO:teuthology.orchestra.run.smithi045.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T03:02:27.387 INFO:teuthology.orchestra.run.smithi045.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T03:02:27.388 INFO:teuthology.orchestra.run.smithi045.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T03:02:27.389 INFO:teuthology.orchestra.run.smithi045.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T03:02:27.389 INFO:teuthology.orchestra.run.smithi045.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T03:02:27.395 INFO:teuthology.orchestra.run.smithi045.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T03:02:27.396 INFO:teuthology.orchestra.run.smithi045.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T03:02:27.412 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T03:02:27.413 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T03:02:27.418 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T03:02:27.422 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T03:02:27.426 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T03:02:27.427 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T03:02:27.428 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T03:02:27.430 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T03:02:27.431 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi045.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T03:02:27.452 INFO:teuthology.orchestra.run.smithi045.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T03:02:27.453 INFO:teuthology.orchestra.run.smithi045.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T03:02:27.496 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T03:02:27.513 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T03:02:27.514 INFO:teuthology.orchestra.run.smithi046.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T03:02:27.515 INFO:teuthology.orchestra.run.smithi046.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T03:02:27.516 INFO:teuthology.orchestra.run.smithi046.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T03:02:27.522 INFO:teuthology.orchestra.run.smithi046.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T03:02:27.523 INFO:teuthology.orchestra.run.smithi046.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T03:02:27.532 INFO:teuthology.orchestra.run.smithi045.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T03:02:27.532 INFO:teuthology.orchestra.run.smithi045.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T03:02:27.535 INFO:teuthology.orchestra.run.smithi045.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T03:02:27.537 INFO:teuthology.orchestra.run.smithi045.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T03:02:27.538 INFO:teuthology.orchestra.run.smithi045.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T03:02:27.548 INFO:teuthology.orchestra.run.smithi045.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T03:02:27.577 INFO:teuthology.orchestra.run.smithi046.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T03:02:27.578 INFO:teuthology.orchestra.run.smithi046.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T03:02:27.580 INFO:teuthology.orchestra.run.smithi046.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T03:02:27.613 INFO:teuthology.orchestra.run.smithi045.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T03:02:27.614 INFO:teuthology.orchestra.run.smithi045.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T03:02:27.615 INFO:teuthology.orchestra.run.smithi045.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T03:02:27.618 INFO:teuthology.orchestra.run.smithi045.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T03:02:27.658 INFO:teuthology.orchestra.run.smithi046.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T03:02:27.659 INFO:teuthology.orchestra.run.smithi046.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T03:02:27.661 INFO:teuthology.orchestra.run.smithi046.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T03:02:27.663 INFO:teuthology.orchestra.run.smithi046.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T03:02:27.664 INFO:teuthology.orchestra.run.smithi046.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T03:02:27.674 INFO:teuthology.orchestra.run.smithi046.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T03:02:27.695 INFO:teuthology.orchestra.run.smithi045.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T03:02:27.696 INFO:teuthology.orchestra.run.smithi045.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T03:02:27.699 INFO:teuthology.orchestra.run.smithi045.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T03:02:27.701 INFO:teuthology.orchestra.run.smithi045.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T03:02:27.705 INFO:teuthology.orchestra.run.smithi045.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T03:02:27.706 INFO:teuthology.orchestra.run.smithi045.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T03:02:27.740 INFO:teuthology.orchestra.run.smithi046.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T03:02:27.741 INFO:teuthology.orchestra.run.smithi046.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T03:02:27.742 INFO:teuthology.orchestra.run.smithi046.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T03:02:27.744 INFO:teuthology.orchestra.run.smithi046.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T03:02:27.776 INFO:teuthology.orchestra.run.smithi045.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T03:02:27.780 INFO:teuthology.orchestra.run.smithi045.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T03:02:27.821 INFO:teuthology.orchestra.run.smithi046.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T03:02:27.823 INFO:teuthology.orchestra.run.smithi046.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T03:02:27.825 INFO:teuthology.orchestra.run.smithi046.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T03:02:27.827 INFO:teuthology.orchestra.run.smithi046.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T03:02:27.832 INFO:teuthology.orchestra.run.smithi046.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T03:02:27.833 INFO:teuthology.orchestra.run.smithi046.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T03:02:27.902 INFO:teuthology.orchestra.run.smithi046.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T03:02:27.907 INFO:teuthology.orchestra.run.smithi046.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T03:02:27.948 INFO:teuthology.orchestra.run.smithi045.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-16T03:02:27.949 INFO:teuthology.orchestra.run.smithi045.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T03:02:27.951 INFO:teuthology.orchestra.run.smithi045.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T03:02:27.953 INFO:teuthology.orchestra.run.smithi045.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T03:02:27.954 INFO:teuthology.orchestra.run.smithi045.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T03:02:27.957 INFO:teuthology.orchestra.run.smithi045.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T03:02:27.994 INFO:teuthology.orchestra.run.smithi045.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T03:02:28.023 INFO:teuthology.orchestra.run.smithi045.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T03:02:28.025 INFO:teuthology.orchestra.run.smithi045.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T03:02:28.047 INFO:teuthology.orchestra.run.smithi045.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-16T03:02:28.053 INFO:teuthology.orchestra.run.smithi045.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-16T03:02:28.053 INFO:teuthology.orchestra.run.smithi045.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T03:02:28.053 INFO:teuthology.orchestra.run.smithi045.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T03:02:28.055 INFO:teuthology.orchestra.run.smithi045.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-16T03:02:28.072 INFO:teuthology.orchestra.run.smithi046.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T03:02:28.074 INFO:teuthology.orchestra.run.smithi046.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T03:02:28.076 INFO:teuthology.orchestra.run.smithi046.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T03:02:28.077 INFO:teuthology.orchestra.run.smithi046.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T03:02:28.079 INFO:teuthology.orchestra.run.smithi046.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T03:02:28.092 INFO:teuthology.orchestra.run.smithi045.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-16T03:02:28.117 INFO:teuthology.orchestra.run.smithi046.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T03:02:28.118 INFO:teuthology.orchestra.run.smithi045.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T03:02:28.119 INFO:teuthology.orchestra.run.smithi045.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T03:02:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T03:02:28.121 INFO:teuthology.orchestra.run.smithi046.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T03:02:28.121 INFO:teuthology.orchestra.run.smithi046.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T03:02:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T03:02:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T03:02:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T03:02:28.130 INFO:teuthology.orchestra.run.smithi045.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T03:02:28.177 INFO:teuthology.orchestra.run.smithi046.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T03:02:28.178 INFO:teuthology.orchestra.run.smithi046.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T03:02:28.199 INFO:teuthology.orchestra.run.smithi045.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T03:02:28.203 INFO:teuthology.orchestra.run.smithi045.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T03:02:28.205 INFO:teuthology.orchestra.run.smithi045.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T03:02:28.240 INFO:teuthology.orchestra.run.smithi046.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T03:02:28.241 INFO:teuthology.orchestra.run.smithi046.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T03:02:28.242 INFO:teuthology.orchestra.run.smithi046.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T03:02:28.243 INFO:teuthology.orchestra.run.smithi046.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T03:02:28.243 INFO:teuthology.orchestra.run.smithi046.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T03:02:28.245 INFO:teuthology.orchestra.run.smithi046.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T03:02:28.251 INFO:teuthology.orchestra.run.smithi046.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T03:02:28.281 INFO:teuthology.orchestra.run.smithi045.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T03:02:28.288 INFO:teuthology.orchestra.run.smithi045.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T03:02:28.290 INFO:teuthology.orchestra.run.smithi045.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T03:02:28.292 INFO:teuthology.orchestra.run.smithi045.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T03:02:28.303 INFO:teuthology.orchestra.run.smithi045.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T03:02:28.304 INFO:teuthology.orchestra.run.smithi045.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T03:02:28.308 INFO:teuthology.orchestra.run.smithi045.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-16T03:02:28.312 INFO:teuthology.orchestra.run.smithi045.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T03:02:28.321 INFO:teuthology.orchestra.run.smithi046.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T03:02:28.340 INFO:teuthology.orchestra.run.smithi046.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T03:02:28.342 INFO:teuthology.orchestra.run.smithi046.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T03:02:28.362 INFO:teuthology.orchestra.run.smithi045.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T03:02:28.364 INFO:teuthology.orchestra.run.smithi045.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T03:02:28.365 INFO:teuthology.orchestra.run.smithi045.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T03:02:28.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T03:02:28.404 INFO:teuthology.orchestra.run.smithi046.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-16T03:02:28.409 INFO:teuthology.orchestra.run.smithi046.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T03:02:28.412 INFO:teuthology.orchestra.run.smithi046.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T03:02:28.413 INFO:teuthology.orchestra.run.smithi046.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T03:02:28.424 INFO:teuthology.orchestra.run.smithi046.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T03:02:28.425 INFO:teuthology.orchestra.run.smithi046.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T03:02:28.433 INFO:teuthology.orchestra.run.smithi046.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T03:02:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T03:02:28.445 INFO:teuthology.orchestra.run.smithi045.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T03:02:28.465 INFO:teuthology.orchestra.run.smithi045.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T03:02:28.466 INFO:teuthology.orchestra.run.smithi045.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T03:02:28.467 INFO:teuthology.orchestra.run.smithi045.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T03:02:28.472 INFO:teuthology.orchestra.run.smithi045.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T03:02:28.474 INFO:teuthology.orchestra.run.smithi045.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T03:02:28.484 INFO:teuthology.orchestra.run.smithi046.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T03:02:28.486 INFO:teuthology.orchestra.run.smithi046.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T03:02:28.486 INFO:teuthology.orchestra.run.smithi046.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-16T03:02:28.489 INFO:teuthology.orchestra.run.smithi046.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-16T03:02:28.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-16T03:02:28.547 INFO:teuthology.orchestra.run.smithi046.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-16T03:02:28.566 INFO:teuthology.orchestra.run.smithi046.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T03:02:28.570 INFO:teuthology.orchestra.run.smithi046.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T03:02:28.572 INFO:teuthology.orchestra.run.smithi046.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T03:02:28.589 INFO:teuthology.orchestra.run.smithi046.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T03:02:28.590 INFO:teuthology.orchestra.run.smithi046.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T03:02:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T03:02:28.596 INFO:teuthology.orchestra.run.smithi046.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T03:02:28.598 INFO:teuthology.orchestra.run.smithi046.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T03:02:28.770 INFO:teuthology.orchestra.run.smithi046.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-16T03:02:29.048 INFO:teuthology.orchestra.run.smithi045.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-16T03:02:29.048 INFO:teuthology.orchestra.run.smithi045.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-16T03:02:29.065 INFO:teuthology.orchestra.run.smithi045.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-16T03:02:29.108 INFO:teuthology.orchestra.run.smithi045.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-16T03:02:29.118 INFO:teuthology.orchestra.run.smithi045.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-16T03:02:29.120 INFO:teuthology.orchestra.run.smithi045.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-16T03:02:29.635 INFO:teuthology.orchestra.run.smithi046.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-16T03:02:29.635 INFO:teuthology.orchestra.run.smithi046.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-16T03:02:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-16T03:02:29.695 INFO:teuthology.orchestra.run.smithi046.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-16T03:02:29.733 INFO:teuthology.orchestra.run.smithi046.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-16T03:02:29.736 INFO:teuthology.orchestra.run.smithi046.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-16T03:02:31.289 INFO:teuthology.orchestra.run.smithi045.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-16T03:02:31.474 INFO:teuthology.orchestra.run.smithi045.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-16T03:02:31.474 INFO:teuthology.orchestra.run.smithi045.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-16T03:02:31.893 INFO:teuthology.orchestra.run.smithi046.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-16T03:02:32.096 INFO:teuthology.orchestra.run.smithi046.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-16T03:02:32.096 INFO:teuthology.orchestra.run.smithi046.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-16T03:02:33.077 INFO:teuthology.orchestra.run.smithi045.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-16T03:02:33.080 INFO:teuthology.orchestra.run.smithi045.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-16T03:02:33.082 INFO:teuthology.orchestra.run.smithi045.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-16T03:02:33.584 INFO:teuthology.orchestra.run.smithi045.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-16T03:02:33.723 INFO:teuthology.orchestra.run.smithi046.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-16T03:02:33.791 INFO:teuthology.orchestra.run.smithi046.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-16T03:02:33.793 INFO:teuthology.orchestra.run.smithi046.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-16T03:02:34.096 INFO:teuthology.orchestra.run.smithi046.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-16T03:02:34.175 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 244 MB in 7s (33.0 MB/s) 2024-09-16T03:02:34.341 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T03:02:34.444 INFO:teuthology.orchestra.run.smithi045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T03:02:34.449 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T03:02:34.531 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:02:34.697 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 244 MB in 8s (31.3 MB/s) 2024-09-16T03:02:34.861 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T03:02:34.901 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T03:02:34.928 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T03:02:34.964 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T03:02:34.967 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:02:34.969 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T03:02:35.052 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:02:35.270 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T03:02:35.294 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T03:02:35.329 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:02:35.397 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T03:02:35.423 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T03:02:35.463 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:02:35.740 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T03:02:35.767 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:02:35.816 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:02:35.841 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T03:02:35.868 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T03:02:35.908 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:02:36.252 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T03:02:36.278 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:02:36.318 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:02:36.361 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T03:02:36.388 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:02:36.436 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:02:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T03:02:36.648 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:02:36.679 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:02:36.823 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T03:02:36.850 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:02:36.890 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:02:37.016 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T03:02:37.043 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T03:02:37.082 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:02:37.201 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T03:02:37.228 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:02:37.267 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:02:37.404 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librados2. 2024-09-16T03:02:37.430 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:37.473 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:37.645 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T03:02:37.672 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T03:02:37.712 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:02:37.898 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libnbd0. 2024-09-16T03:02:37.914 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-16T03:02:37.947 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-16T03:02:38.040 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librados2. 2024-09-16T03:02:38.067 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:38.107 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:38.267 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librbd1. 2024-09-16T03:02:38.294 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:38.333 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:38.518 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libnbd0. 2024-09-16T03:02:38.545 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-16T03:02:38.576 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-16T03:02:38.887 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librbd1. 2024-09-16T03:02:38.914 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:38.954 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:39.011 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T03:02:39.038 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:39.078 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:39.355 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rados. 2024-09-16T03:02:39.371 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:39.405 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:39.607 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T03:02:39.634 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:39.666 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T03:02:39.673 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:39.693 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:02:39.732 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:39.909 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rados. 2024-09-16T03:02:39.925 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:39.959 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:39.976 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T03:02:40.005 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:40.035 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:40.270 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T03:02:40.297 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:02:40.337 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:40.337 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T03:02:40.364 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:02:40.404 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:40.589 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T03:02:40.605 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:40.648 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:40.723 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T03:02:40.750 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T03:02:40.790 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:02:40.984 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T03:02:41.011 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:02:41.050 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:41.109 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T03:02:41.136 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T03:02:41.176 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T03:02:41.378 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T03:02:41.405 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T03:02:41.437 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T03:02:41.445 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:02:41.453 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:41.487 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:41.748 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T03:02:41.775 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T03:02:41.781 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T03:02:41.797 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T03:02:41.806 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T03:02:41.831 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:02:42.059 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T03:02:42.082 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:42.108 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T03:02:42.124 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T03:02:42.125 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:42.166 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:02:42.478 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T03:02:42.478 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T03:02:42.504 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T03:02:42.505 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T03:02:42.545 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:02:42.553 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:02:42.839 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T03:02:42.855 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T03:02:42.881 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua5.1. 2024-09-16T03:02:42.898 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:02:42.907 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T03:02:42.947 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:02:43.167 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T03:02:43.183 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T03:02:43.217 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua-any. 2024-09-16T03:02:43.217 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:02:43.232 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-16T03:02:43.267 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T03:02:43.545 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package lua5.1. 2024-09-16T03:02:43.561 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package zip. 2024-09-16T03:02:43.572 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T03:02:43.577 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-16T03:02:43.612 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:02:43.619 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T03:02:43.932 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package lua-any. 2024-09-16T03:02:43.939 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package unzip. 2024-09-16T03:02:43.947 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-16T03:02:43.966 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T03:02:43.998 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T03:02:44.005 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:02:44.268 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package zip. 2024-09-16T03:02:44.294 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-16T03:02:44.376 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package luarocks. 2024-09-16T03:02:44.376 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T03:02:44.402 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T03:02:44.442 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:02:44.688 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package unzip. 2024-09-16T03:02:44.714 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T03:02:44.754 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:02:44.858 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librgw2. 2024-09-16T03:02:44.885 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:44.924 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:45.075 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package luarocks. 2024-09-16T03:02:45.102 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T03:02:45.133 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:02:45.491 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librgw2. 2024-09-16T03:02:45.518 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:45.535 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T03:02:45.557 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:45.562 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:45.601 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:45.888 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T03:02:45.905 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T03:02:45.939 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:02:46.200 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T03:02:46.200 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:46.210 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:46.257 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T03:02:46.284 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:46.324 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:46.554 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T03:02:46.581 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T03:02:46.621 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:02:46.760 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-common. 2024-09-16T03:02:46.787 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:46.826 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:46.882 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T03:02:46.898 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:46.941 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:47.402 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-common. 2024-09-16T03:02:47.429 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:47.469 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:48.290 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-base. 2024-09-16T03:02:48.317 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:48.357 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:48.919 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T03:02:48.946 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T03:02:48.985 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:02:49.049 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-base. 2024-09-16T03:02:49.076 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:02:49.126 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:02:49.538 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T03:02:49.568 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T03:02:49.604 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:02:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T03:02:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T03:02:49.916 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T03:02:49.920 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:02:49.943 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T03:02:49.982 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:02:50.223 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T03:02:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T03:02:50.260 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T03:02:50.287 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T03:02:50.289 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:02:50.326 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:02:50.588 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T03:02:50.609 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T03:02:50.614 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T03:02:50.636 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T03:02:50.654 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:02:50.676 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:02:50.957 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T03:02:50.978 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T03:02:50.983 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T03:02:51.005 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T03:02:51.023 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T03:02:51.045 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:02:51.323 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T03:02:51.326 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-portend. 2024-09-16T03:02:51.350 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T03:02:51.353 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-16T03:02:51.381 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:02:51.392 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T03:02:51.662 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T03:02:51.689 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T03:02:51.701 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T03:02:51.720 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T03:02:51.728 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T03:02:51.767 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T03:02:51.998 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T03:02:52.025 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T03:02:52.065 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:02:52.070 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-portend. 2024-09-16T03:02:52.097 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-16T03:02:52.137 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T03:02:52.392 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T03:02:52.408 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T03:02:52.442 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T03:02:52.448 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T03:02:52.475 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T03:02:52.515 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T03:02:52.704 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T03:02:52.731 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T03:02:52.762 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T03:02:52.776 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T03:02:52.803 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T03:02:52.834 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:02:53.048 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T03:02:53.075 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T03:02:53.114 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:02:53.171 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T03:02:53.198 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T03:02:53.237 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T03:02:53.417 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-mako. 2024-09-16T03:02:53.444 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T03:02:53.483 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T03:02:53.509 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T03:02:53.536 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:02:53.541 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T03:02:53.843 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T03:02:53.862 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T03:02:53.870 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T03:02:53.889 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T03:02:53.910 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:02:53.937 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:02:54.198 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T03:02:54.214 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T03:02:54.238 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-mako. 2024-09-16T03:02:54.247 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:02:54.265 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T03:02:54.313 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:02:54.550 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-webob. 2024-09-16T03:02:54.577 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T03:02:54.617 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:02:54.641 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T03:02:54.668 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T03:02:54.708 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:02:54.945 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T03:02:54.960 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T03:02:54.972 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T03:02:54.976 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T03:02:55.010 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:02:55.014 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:02:55.305 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webob. 2024-09-16T03:02:55.322 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T03:02:55.332 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T03:02:55.349 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T03:02:55.371 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:02:55.389 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:02:55.692 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-paste. 2024-09-16T03:02:55.719 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T03:02:55.724 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T03:02:55.752 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T03:02:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:02:55.794 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:02:56.094 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T03:02:56.102 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T03:02:56.110 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T03:02:56.130 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T03:02:56.152 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:02:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:02:56.438 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-paste. 2024-09-16T03:02:56.455 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T03:02:56.465 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T03:02:56.483 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T03:02:56.496 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:02:56.522 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:02:56.783 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T03:02:56.810 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T03:02:56.825 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T03:02:56.841 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T03:02:56.853 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T03:02:56.891 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:02:57.111 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T03:02:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T03:02:57.153 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T03:02:57.177 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:02:57.179 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T03:02:57.211 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:02:57.522 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T03:02:57.523 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T03:02:57.550 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T03:02:57.551 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T03:02:57.589 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T03:02:57.590 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T03:02:57.900 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T03:02:57.928 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T03:02:57.951 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-idna. 2024-09-16T03:02:57.967 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:02:57.978 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T03:02:58.017 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:02:58.330 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T03:02:58.345 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T03:02:58.357 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T03:02:58.372 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T03:02:58.396 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T03:02:58.412 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:02:58.715 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-requests. 2024-09-16T03:02:58.724 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-idna. 2024-09-16T03:02:58.742 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T03:02:58.752 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T03:02:58.781 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:02:58.791 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:02:59.060 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T03:02:59.077 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T03:02:59.092 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T03:02:59.110 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:02:59.119 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T03:02:59.159 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:02:59.380 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-requests. 2024-09-16T03:02:59.408 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T03:02:59.797 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:03:00.002 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T03:03:00.029 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:00.512 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:00.717 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T03:03:00.745 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T03:03:00.808 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:03:00.884 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T03:03:00.911 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:00.950 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:01.070 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T03:03:01.087 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:01.128 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:01.219 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T03:03:01.246 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T03:03:01.286 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:03:01.511 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T03:03:01.538 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:01.556 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T03:03:01.577 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:01.583 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T03:03:01.622 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T03:03:01.888 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T03:03:01.915 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T03:03:01.954 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:03:01.958 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T03:03:01.986 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:02.024 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:02.240 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T03:03:02.267 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T03:03:02.306 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T03:03:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T03:03:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:02.618 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T03:03:02.646 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:02.676 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:03.046 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T03:03:03.073 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:03.112 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:03.122 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T03:03:03.149 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:03.180 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:03.807 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T03:03:03.834 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:03.874 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:04.350 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph. 2024-09-16T03:03:04.367 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:04.408 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:04.727 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T03:03:04.754 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:04.794 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:04.918 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph. 2024-09-16T03:03:04.935 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:04.977 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:05.205 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T03:03:05.233 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:05.289 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:05.296 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T03:03:05.323 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:05.362 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:05.715 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T03:03:05.743 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:05.775 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package cephadm. 2024-09-16T03:03:05.782 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:05.802 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:05.849 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:06.177 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T03:03:06.185 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package cephadm. 2024-09-16T03:03:06.205 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T03:03:06.212 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:06.243 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T03:03:06.251 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:06.579 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T03:03:06.606 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T03:03:06.622 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T03:03:06.645 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T03:03:06.649 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:03:06.696 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:03:06.990 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T03:03:07.017 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:03:07.056 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:03:07.668 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-babel. 2024-09-16T03:03:07.695 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:03:07.725 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:03:08.028 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T03:03:08.056 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T03:03:08.086 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:03:08.094 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-babel. 2024-09-16T03:03:08.111 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:03:08.144 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:03:08.390 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T03:03:08.417 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:08.447 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T03:03:08.447 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:08.474 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T03:03:08.513 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:03:08.759 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T03:03:08.787 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T03:03:08.817 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T03:03:08.841 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T03:03:08.869 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:08.908 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:09.062 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-routes. 2024-09-16T03:03:09.089 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T03:03:09.128 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:03:09.228 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T03:03:09.244 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T03:03:09.278 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T03:03:09.448 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T03:03:09.475 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:09.506 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:09.522 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-routes. 2024-09-16T03:03:09.550 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T03:03:09.580 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:03:09.909 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T03:03:09.936 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:09.966 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:12.493 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T03:03:12.511 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T03:03:12.551 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:03:13.063 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T03:03:13.091 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T03:03:13.121 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:03:13.163 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T03:03:13.193 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T03:03:13.238 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:03:13.483 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T03:03:13.511 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T03:03:13.549 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:03:13.741 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T03:03:13.769 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T03:03:13.807 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:03:13.852 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T03:03:13.880 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T03:03:14.012 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:03:14.169 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T03:03:14.197 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T03:03:14.236 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:03:14.547 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T03:03:14.575 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T03:03:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T03:03:14.691 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:14.707 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:03:14.729 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:15.425 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T03:03:15.453 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:15.491 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:15.699 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T03:03:15.727 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T03:03:15.765 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T03:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T03:03:16.097 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T03:03:16.134 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:03:16.463 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T03:03:16.491 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T03:03:16.511 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T03:03:16.529 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:03:16.539 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T03:03:16.577 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T03:03:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T03:03:16.869 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-16T03:03:16.880 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T03:03:16.907 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T03:03:16.908 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T03:03:16.946 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:03:17.194 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T03:03:17.210 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T03:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T03:03:17.275 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T03:03:17.303 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T03:03:17.366 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:03:17.557 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T03:03:17.586 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T03:03:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:03:17.678 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T03:03:17.706 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-16T03:03:17.753 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T03:03:17.935 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T03:03:17.963 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T03:03:18.001 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T03:03:18.031 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T03:03:18.059 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T03:03:18.097 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T03:03:18.322 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T03:03:18.350 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T03:03:18.375 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T03:03:18.388 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:03:18.394 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T03:03:18.425 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:03:18.729 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T03:03:18.757 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T03:03:18.795 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T03:03:19.110 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T03:03:19.127 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:19.168 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:19.173 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T03:03:19.201 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T03:03:19.248 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:03:19.455 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T03:03:19.483 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T03:03:19.513 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:03:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T03:03:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:03:19.895 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T03:03:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:03:19.912 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:19.953 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:20.169 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package jq. 2024-09-16T03:03:20.197 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:03:20.223 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T03:03:20.235 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T03:03:20.251 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T03:03:20.289 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:03:20.571 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package socat. 2024-09-16T03:03:20.600 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T03:03:20.637 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:03:20.801 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T03:03:20.829 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:03:20.867 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:03:20.941 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T03:03:20.958 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T03:03:20.991 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:03:21.136 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package jq. 2024-09-16T03:03:21.153 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:03:21.194 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T03:03:21.337 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-test. 2024-09-16T03:03:21.354 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:21.395 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:21.439 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package socat. 2024-09-16T03:03:21.455 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T03:03:21.488 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:03:21.758 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T03:03:21.775 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T03:03:21.808 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:03:22.162 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-test. 2024-09-16T03:03:22.179 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:22.221 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:23.675 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T03:03:23.703 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:23.741 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:24.053 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T03:03:24.082 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:24.119 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:24.439 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T03:03:24.467 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T03:03:24.483 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T03:03:24.500 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T03:03:24.505 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:03:24.533 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:24.793 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T03:03:24.810 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T03:03:24.837 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T03:03:24.843 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:03:24.854 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:24.887 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:25.190 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T03:03:25.218 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T03:03:25.222 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T03:03:25.250 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T03:03:25.256 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:03:25.288 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:03:25.594 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T03:03:25.622 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T03:03:25.660 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:03:25.712 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package pkg-config. 2024-09-16T03:03:25.739 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T03:03:25.777 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:03:26.031 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T03:03:26.038 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T03:03:26.059 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T03:03:26.067 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T03:03:26.097 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:03:26.104 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:03:26.529 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package pkg-config. 2024-09-16T03:03:26.533 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T03:03:26.556 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T03:03:26.562 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T03:03:26.594 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:03:26.599 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T03:03:26.821 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T03:03:26.835 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T03:03:26.860 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T03:03:26.879 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:03:26.889 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T03:03:26.918 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T03:03:27.197 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T03:03:27.226 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T03:03:27.263 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:03:27.282 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T03:03:27.311 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T03:03:27.357 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T03:03:27.575 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T03:03:27.604 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T03:03:27.641 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T03:03:27.660 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T03:03:27.689 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T03:03:27.735 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T03:03:27.978 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-py. 2024-09-16T03:03:28.006 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-16T03:03:28.044 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T03:03:28.055 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T03:03:28.083 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T03:03:28.121 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:03:28.389 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T03:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T03:03:28.432 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T03:03:28.456 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:03:28.461 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T03:03:28.499 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T03:03:28.843 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-py. 2024-09-16T03:03:28.872 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-16T03:03:28.910 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T03:03:28.918 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T03:03:28.939 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T03:03:28.976 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:03:29.197 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T03:03:29.226 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T03:03:29.255 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:03:29.271 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-toml. 2024-09-16T03:03:29.300 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-16T03:03:29.337 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T03:03:29.573 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T03:03:29.590 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T03:03:29.631 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:03:29.718 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T03:03:29.747 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T03:03:29.784 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:03:29.944 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T03:03:29.973 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T03:03:30.010 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:03:30.070 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-toml. 2024-09-16T03:03:30.099 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-16T03:03:30.137 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T03:03:30.271 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T03:03:30.288 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T03:03:30.321 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:03:30.439 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T03:03:30.468 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T03:03:30.505 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:03:30.817 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T03:03:30.846 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T03:03:30.875 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:03:31.017 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package radosgw. 2024-09-16T03:03:31.046 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:31.083 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:31.128 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T03:03:31.145 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T03:03:31.186 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:03:31.882 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package radosgw. 2024-09-16T03:03:31.911 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:31.948 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:32.003 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T03:03:32.020 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:32.061 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:32.367 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T03:03:32.616 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:03:32.742 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:03:32.926 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T03:03:32.943 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T03:03:32.980 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T03:03:32.984 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:33.299 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T03:03:33.306 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T03:03:33.564 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:03:33.690 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:03:33.715 INFO:teuthology.orchestra.run.smithi045.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T03:03:33.785 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:33.956 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T03:03:34.299 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T03:03:34.310 INFO:teuthology.orchestra.run.smithi045.stdout:Adding system user cephadm....done 2024-09-16T03:03:34.467 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:03:34.726 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:03:34.750 INFO:teuthology.orchestra.run.smithi046.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T03:03:34.808 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:34.977 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:03:35.103 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:03:35.380 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T03:03:35.433 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user cephadm....done 2024-09-16T03:03:35.589 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:03:35.639 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:03:35.765 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T03:03:35.890 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:03:36.067 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:03:36.142 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:03:36.243 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:03:36.427 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T03:03:36.544 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T03:03:36.695 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T03:03:36.803 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:03:36.929 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T03:03:36.988 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:03:37.239 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:03:37.273 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:03:37.533 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:03:37.624 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T03:03:37.792 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:03:37.859 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T03:03:38.066 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:38.127 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:03:38.377 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:03:38.378 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:03:38.503 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-16T03:03:38.629 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:03:38.630 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:03:38.764 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:03:38.881 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:03:38.897 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:03:39.149 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:03:39.232 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:39.508 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:03:39.508 INFO:teuthology.orchestra.run.smithi045.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T03:03:39.530 INFO:teuthology.orchestra.run.smithi045.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T03:03:39.586 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:03:39.625 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-16T03:03:39.703 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T03:03:39.726 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:03:39.827 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:03:39.953 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:03:40.071 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:03:40.197 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T03:03:40.221 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:03:40.465 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:03:40.624 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T03:03:40.684 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T03:03:40.724 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:03:40.749 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:03:40.866 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T03:03:41.042 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T03:03:41.143 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:03:41.226 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:03:41.327 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T03:03:41.586 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:03:41.837 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:03:41.854 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:03:42.096 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:03:42.155 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T03:03:42.222 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:03:42.281 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:03:42.348 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:03:42.649 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T03:03:43.016 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:03:43.042 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T03:03:43.267 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:03:43.285 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:03:43.368 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:03:43.695 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:03:43.738 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:03:43.996 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:03:44.120 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T03:03:44.356 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:03:44.580 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T03:03:44.649 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:03:44.758 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:44.839 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:03:45.059 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:03:45.224 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:03:45.385 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:03:45.517 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:03:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:03:46.103 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:03:46.229 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:46.547 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:03:46.642 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:03:46.860 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:03:46.873 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:03:46.977 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T03:03:47.245 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:03:47.371 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:03:47.497 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:03:47.815 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:03:48.072 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:03:48.075 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T03:03:48.324 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:03:48.326 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:03:48.427 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:03:48.457 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T03:03:48.720 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:03:48.725 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:03:48.843 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:03:48.969 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:03:48.979 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T03:03:49.247 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T03:03:49.279 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:03:49.515 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:03:49.555 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T03:03:49.766 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T03:03:49.822 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:03:49.948 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:03:50.034 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:03:50.160 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T03:03:50.250 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:03:50.427 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:03:50.509 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T03:03:50.553 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:03:50.760 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T03:03:50.888 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:03:51.019 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:03:51.189 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:03:51.270 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T03:03:51.440 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:03:51.522 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:03:51.566 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:03:51.648 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T03:03:51.675 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:03:51.747 INFO:teuthology.orchestra.run.smithi045.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T03:03:51.949 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:03:51.985 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:03:52.067 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:03:52.111 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:03:52.351 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:03:52.521 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:03:52.660 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:03:52.781 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T03:03:52.882 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T03:03:52.928 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:03:53.037 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:03:53.125 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:03:53.155 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:03:53.235 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T03:03:53.250 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:03:53.473 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:03:53.585 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:03:53.599 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:03:53.886 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:03:53.984 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:03:54.171 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T03:03:54.227 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T03:03:54.296 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T03:03:54.353 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T03:03:54.581 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T03:03:54.612 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:03:54.746 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:03:54.951 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:03:55.073 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:03:55.202 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:03:55.398 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:03:55.587 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T03:03:55.683 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T03:03:55.792 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T03:03:55.872 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:03:56.051 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T03:03:56.208 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:03:56.309 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:56.405 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:03:56.410 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:56.519 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:03:56.664 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:03:56.856 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:56.982 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:57.083 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T03:03:57.116 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T03:03:57.367 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:03:57.393 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:57.519 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:57.645 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:57.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:03:57.864 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:03:57.864 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:03:57.879 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:58.013 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:58.139 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:03:58.467 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:58.555 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:58.576 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:58.686 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T03:03:58.764 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:58.890 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:58.962 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:59.008 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:03:59.223 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:59.389 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:03:59.611 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:03:59.612 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:04:00.124 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:00.250 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:00.261 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:00.376 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:00.387 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:00.502 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:04:00.505 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:00.606 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:00.707 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:01.233 INFO:teuthology.orchestra.run.smithi045.stdout:Adding group ceph....done 2024-09-16T03:04:01.560 INFO:teuthology.orchestra.run.smithi045.stdout:Adding system user ceph....done 2024-09-16T03:04:01.704 INFO:teuthology.orchestra.run.smithi045.stdout:Setting system user ceph properties....done 2024-09-16T03:04:01.720 INFO:teuthology.orchestra.run.smithi045.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T03:04:01.806 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:01.871 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T03:04:01.931 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:02.057 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:02.183 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:02.244 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T03:04:02.309 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:02.717 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:02.837 INFO:teuthology.orchestra.run.smithi046.stdout:Adding group ceph....done 2024-09-16T03:04:02.843 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:03.170 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user ceph....done 2024-09-16T03:04:03.304 INFO:teuthology.orchestra.run.smithi046.stdout:Setting system user ceph properties....done 2024-09-16T03:04:03.315 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:04:03.315 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:04:03.318 INFO:teuthology.orchestra.run.smithi046.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T03:04:03.463 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T03:04:03.753 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:03.792 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T03:04:04.117 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T03:04:04.260 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:04.445 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:04.571 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:04.804 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:04:04.805 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:04:04.922 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:04:04.922 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:04:05.282 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:05.440 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:04:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:04:05.769 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T03:04:05.985 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:06.224 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:06.255 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:04:06.255 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:04:06.458 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:04:06.458 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:04:06.703 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:06.829 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:06.917 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:06.964 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:07.147 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:04:07.147 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:04:07.163 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:04:07.163 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:04:07.607 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:07.618 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:07.750 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:07.867 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:07.904 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:04:07.904 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:04:08.002 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:08.362 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:08.363 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T03:04:08.471 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:08.472 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T03:04:08.614 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:08.616 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T03:04:08.824 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:04:08.824 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:04:09.273 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:09.441 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:09.727 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T03:04:09.747 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:09.888 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T03:04:10.266 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T03:04:10.367 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T03:04:10.392 INFO:teuthology.orchestra.run.smithi045.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T03:04:10.399 DEBUG:teuthology.parallel:result is None 2024-09-16T03:04:10.519 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T03:04:11.528 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T03:04:12.177 INFO:teuthology.orchestra.run.smithi046.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T03:04:12.182 DEBUG:teuthology.parallel:result is None 2024-09-16T03:04:12.183 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:04:12.328 DEBUG:teuthology.orchestra.run.smithi045:> dpkg-query -W -f '${Version}' ceph 2024-09-16T03:04:12.348 INFO:teuthology.orchestra.run.smithi045.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T03:04:12.348 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T03:04:12.349 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-16T03:04:12.351 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:04:12.488 DEBUG:teuthology.orchestra.run.smithi046:> dpkg-query -W -f '${Version}' ceph 2024-09-16T03:04:12.507 INFO:teuthology.orchestra.run.smithi046.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T03:04:12.507 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T03:04:12.508 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-16T03:04:12.511 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T03:04:12.511 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:12.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:04:12.527 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:12.527 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:04:12.563 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T03:04:12.563 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:12.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:04:12.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:04:12.636 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:12.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:04:12.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:04:12.702 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T03:04:12.702 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:12.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:04:12.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:04:12.772 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:12.772 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:04:12.786 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:04:12.841 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T03:04:12.842 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:12.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:04:12.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:04:12.909 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:12.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:04:12.922 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:04:12.977 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T03:04:13.065 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T03:04:13.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /var/log/ceph 2024-09-16T03:04:13.068 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2024-09-16T03:04:13.081 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T03:04:13.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:04:13.122 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:04:13.137 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T03:04:13.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:04:13.178 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:04:13.203 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T03:04:13.203 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-16T03:04:13.203 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823', 'branch': 'main', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905823', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 2855, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'sleep_before_teardown': 0, 'subset': '17936/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7498493279982dfe87f00616198a5967475f6169', 'targets': {'smithi045.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP5pH6b1OqOYcSULw3nxTnOslOOs/XDZOgoqzzq6yKSlpb1pXa/bZ43/JPENvQ9V4I/dIR1pfVmYEMQ1vUke5G0=', 'smithi046.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC7ukruT5mdjQVQQe3hkdNsgSc7XyHVGPdggN4PfWH53wxvE+xIiEcKKO8H8wvhO/yoCxz1U3Iusi9WsfFvUkiQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_20:00:16', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T03:04:13.203 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:04:13.234 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:04:13.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:04:13.282 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:04:13.299 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:13.299 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:04:13.342 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:04:13.342 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-09-16T03:04:13.389 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:04:13.390 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:04:13.390 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-09-16T03:04:13.390 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:04:13.390 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-16 02:58:42.037834526 +0000 2024-09-16T03:04:13.390 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-16 02:58:41.773836884 +0000 2024-09-16T03:04:13.390 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-16 02:58:41.773836884 +0000 2024-09-16T03:04:13.390 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-09-16T03:04:13.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:04:13.444 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-16T03:04:13.444 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-16T03:04:13.444 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000315301 s, 1.6 MB/s 2024-09-16T03:04:13.446 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:04:13.495 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-09-16T03:04:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:04:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:04:13.546 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-09-16T03:04:13.546 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:04:13.546 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-16 02:58:42.493830453 +0000 2024-09-16T03:04:13.546 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-16 02:58:42.253832597 +0000 2024-09-16T03:04:13.546 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-16 02:58:42.253832597 +0000 2024-09-16T03:04:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-09-16T03:04:13.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:04:13.601 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-16T03:04:13.601 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-16T03:04:13.601 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000341137 s, 1.5 MB/s 2024-09-16T03:04:13.602 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:04:13.651 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-09-16T03:04:13.698 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:04:13.698 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:04:13.698 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-09-16T03:04:13.698 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:04:13.698 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-16 02:58:43.037825593 +0000 2024-09-16T03:04:13.698 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-16 02:58:42.741828237 +0000 2024-09-16T03:04:13.698 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-16 02:58:42.741828237 +0000 2024-09-16T03:04:13.698 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-09-16T03:04:13.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:04:13.752 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-16T03:04:13.752 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-16T03:04:13.752 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000299307 s, 1.7 MB/s 2024-09-16T03:04:13.753 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:04:13.803 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-09-16T03:04:13.849 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:04:13.849 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:04:13.850 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 939 Links: 1 2024-09-16T03:04:13.850 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:04:13.850 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-16 02:58:43.549821021 +0000 2024-09-16T03:04:13.850 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-16 02:58:43.277823450 +0000 2024-09-16T03:04:13.850 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-16 02:58:43.277823450 +0000 2024-09-16T03:04:13.850 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-09-16T03:04:13.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:04:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-16T03:04:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-16T03:04:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000296353 s, 1.7 MB/s 2024-09-16T03:04:13.906 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:04:13.955 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T03:04:13.955 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:13.955 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:04:13.961 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:04:13.961 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-09-16T03:04:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:04:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:04:14.007 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-09-16T03:04:14.007 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:04:14.007 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-09-16 02:58:40.773704840 +0000 2024-09-16T03:04:14.007 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-09-16 02:58:40.513707151 +0000 2024-09-16T03:04:14.007 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-09-16 02:58:40.513707151 +0000 2024-09-16T03:04:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-09-16T03:04:14.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:04:14.061 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-09-16T03:04:14.061 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-09-16T03:04:14.061 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000250846 s, 2.0 MB/s 2024-09-16T03:04:14.063 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:04:14.112 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-09-16T03:04:14.158 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:04:14.159 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:04:14.159 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-09-16T03:04:14.159 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:04:14.159 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-09-16 02:58:41.253700573 +0000 2024-09-16T03:04:14.159 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-09-16 02:58:41.001702812 +0000 2024-09-16T03:04:14.159 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-09-16 02:58:41.001702812 +0000 2024-09-16T03:04:14.159 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-09-16T03:04:14.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:04:14.213 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-09-16T03:04:14.213 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-09-16T03:04:14.213 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000311966 s, 1.6 MB/s 2024-09-16T03:04:14.214 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:04:14.264 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-09-16T03:04:14.311 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:04:14.311 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:04:14.311 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-09-16T03:04:14.311 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:04:14.311 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-09-16 02:58:41.757696093 +0000 2024-09-16T03:04:14.311 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-09-16 02:58:41.501698369 +0000 2024-09-16T03:04:14.311 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-09-16 02:58:41.501698369 +0000 2024-09-16T03:04:14.311 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-09-16T03:04:14.312 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:04:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-09-16T03:04:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-09-16T03:04:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000199606 s, 2.6 MB/s 2024-09-16T03:04:14.366 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:04:14.417 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-09-16T03:04:14.462 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:04:14.463 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:04:14.463 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 944 Links: 1 2024-09-16T03:04:14.463 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:04:14.463 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-09-16 02:58:42.253691684 +0000 2024-09-16T03:04:14.463 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-09-16 02:58:41.985694067 +0000 2024-09-16T03:04:14.463 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-09-16 02:58:41.985694067 +0000 2024-09-16T03:04:14.463 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-09-16T03:04:14.463 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:04:14.518 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-09-16T03:04:14.518 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-09-16T03:04:14.518 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000334242 s, 1.5 MB/s 2024-09-16T03:04:14.519 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:04:14.568 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T03:04:14.568 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T03:04:14.568 INFO:tasks.ceph:Generating config... 2024-09-16T03:04:14.569 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T03:04:14.569 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T03:04:14.569 INFO:tasks.ceph:[global] debug ms = 1 2024-09-16T03:04:14.569 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-16T03:04:14.569 INFO:tasks.ceph:[global] debug rados = 20 2024-09-16T03:04:14.569 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T03:04:14.569 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T03:04:14.570 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-16T03:04:14.570 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T03:04:14.570 INFO:tasks.ceph:[global] ms type = async 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T03:04:14.570 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T03:04:14.571 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T03:04:14.571 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-16T03:04:14.571 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T03:04:14.571 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T03:04:14.572 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T03:04:14.573 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T03:04:14.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T03:04:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T03:04:14.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T03:04:14.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:04:14.749 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.45:6789]'), ('mon.c', '[v1:172.21.15.45:6790]'), ('mon.b', '[v1:172.21.15.46:6789]')] 2024-09-16T03:04:14.749 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.45:6789],[v1:172.21.15.45:6790],[v1:172.21.15.46:6789]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T03:04:14.750 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:14.750 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:04:14.797 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.45:6789]' --addv c '[v1:172.21.15.45:6790]' --addv b '[v1:172.21.15.46:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:14.885 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:14.885 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: generated fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:14.885 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = quincy 2024-09-16T03:04:14.885 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 0 2024-09-16T03:04:14.885 INFO:teuthology.orchestra.run.smithi045.stdout:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:14.885 INFO:teuthology.orchestra.run.smithi045.stdout:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:14.885 INFO:teuthology.orchestra.run.smithi045.stdout:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:14.885 INFO:teuthology.orchestra.run.smithi045.stdout:min_mon_release 17 (quincy) 2024-09-16T03:04:14.886 INFO:teuthology.orchestra.run.smithi045.stdout:election_strategy: 1 2024-09-16T03:04:14.886 INFO:teuthology.orchestra.run.smithi045.stdout:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:14.886 INFO:teuthology.orchestra.run.smithi045.stdout:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:14.886 INFO:teuthology.orchestra.run.smithi045.stdout:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:14.886 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T03:04:14.887 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:04:14.891 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 16b8bfd8-03c6-4432-90b7-8e9932b37929... 2024-09-16T03:04:14.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:04:14.934 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:04:14.962 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2024-09-16T03:04:14.963 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2024-09-16T03:04:14.964 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2024-09-16T03:04:14.965 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = "[v1:172.21.15.45:6789],[v1:172.21.15.45:6790],[v1:172.21.15.46:6789]" 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: debug objecter = 20 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: debug rados = 20 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: mon client directed command retry = 5 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: mon election default strategy = 1 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: ms bind msgr2 = False 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: ms inject socket failures = 5000 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: ms type = async 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2024-09-16T03:04:14.966 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 600 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug trim objects = true 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2024-09-16T03:04:14.967 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: bdev async discard = True 2024-09-16T03:04:14.968 INFO:teuthology.orchestra.run.smithi045.stdout: bdev enable discard = True 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore allocator = bitmap 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore block size = 96636764160 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore fsck on mount = True 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore write v2 = False 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore zero block detection = True 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluefs = 20 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluestore = 20 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: debug rocksdb = 10 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd full ratio = 0.9 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:04:14.969 INFO:teuthology.orchestra.run.smithi045.stdout: osd class default list = * 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: osd class load list = * 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify cached snaps = True 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify missing on start = True 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock override recovery settings = True 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: osd objectstore = bluestore 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue = debug_random 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue cut off = debug_random 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-09-16T03:04:14.970 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2024-09-16T03:04:14.971 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: # 1m isn't quite enough 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:04:14.972 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: mon min osdmap epochs = 25 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune interval = 2 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune min = 15 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: mon scrub inject missing keys = 0.05 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: mon scrub interval = 300 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on pool no app = False 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: paxos service trim min = 5 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2024-09-16T03:04:14.973 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2024-09-16T03:04:14.974 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:04:14.974 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:04:14.974 INFO:teuthology.orchestra.run.smithi045.stdout: debug monc = 20 2024-09-16T03:04:14.974 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-09-16T03:04:14.974 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2024-09-16T03:04:14.974 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.c] 2024-09-16T03:04:14.974 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.b] 2024-09-16T03:04:14.974 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-09-16T03:04:14.974 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:04:14.975 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2024-09-16T03:04:14.976 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2024-09-16T03:04:14.977 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "[v1:172.21.15.45:6789],[v1:172.21.15.45:6790],[v1:172.21.15.46:6789]" 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: debug objecter = 20 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: debug rados = 20 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: mon client directed command retry = 5 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: mon election default strategy = 1 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: ms bind msgr2 = False 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject socket failures = 5000 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: ms type = async 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2024-09-16T03:04:14.978 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug trim objects = true 2024-09-16T03:04:14.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.980 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: bdev async discard = True 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: bdev enable discard = True 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore allocator = bitmap 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore write v2 = False 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore zero block detection = True 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 20 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 20 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 10 2024-09-16T03:04:14.981 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: osd class default list = * 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: osd class load list = * 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify cached snaps = True 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify missing on start = True 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock override recovery settings = True 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: osd op queue = debug_random 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: osd op queue cut off = debug_random 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2024-09-16T03:04:14.982 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2024-09-16T03:04:14.983 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.984 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon min osdmap epochs = 25 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon osdmap full prune interval = 2 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon osdmap full prune min = 15 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon scrub inject missing keys = 0.05 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon scrub interval = 300 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on pool no app = False 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: paxos service trim min = 5 2024-09-16T03:04:14.985 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout: debug monc = 20 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2024-09-16T03:04:14.986 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2024-09-16T03:04:14.987 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T03:04:14.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T03:04:15.078 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T03:04:15.078 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:15.078 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T03:04:15.083 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:15.083 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T03:04:15.129 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2024-09-16T03:04:15.129 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:15.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:04:15.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:04:15.194 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:15.195 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:15.241 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T03:04:15.242 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:15.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:04:15.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:04:15.267 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:15.267 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:15.314 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T03:04:15.315 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T03:04:15.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:04:15.404 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:04:15.418 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T03:04:15.418 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T03:04:15.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T03:04:15.452 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T03:04:15.462 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T03:04:15.463 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T03:04:15.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T03:04:15.479 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T03:04:15.479 INFO:tasks.ceph:role: osd.0 2024-09-16T03:04:15.479 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-16T03:04:15.479 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:04:15.535 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:04:15.535 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:04:15.536 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:04:15.536 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:04:15.536 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:04:15.537 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:04:15.537 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:04:15.537 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:04:15.537 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:04:15.537 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:04:15.557 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-09-16T03:04:15.559 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-09-16T03:04:15.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T03:04:15.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T03:04:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:04:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:04:15.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T03:04:15.724 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T03:04:15.724 INFO:tasks.ceph:role: osd.1 2024-09-16T03:04:15.724 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-16T03:04:15.724 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:04:15.777 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:04:15.777 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:04:15.777 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:04:15.777 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:04:15.777 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:04:15.777 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:04:15.777 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:04:15.777 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:04:15.778 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:04:15.778 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:04:15.801 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-09-16T03:04:15.802 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-09-16T03:04:15.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T03:04:15.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T03:04:15.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:04:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:04:15.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T03:04:15.967 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T03:04:15.967 INFO:tasks.ceph:role: osd.2 2024-09-16T03:04:15.967 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-16T03:04:15.967 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:04:16.021 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:04:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:04:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:04:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:04:16.021 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:04:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:04:16.022 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:04:16.022 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:04:16.022 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:04:16.022 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:04:16.045 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-09-16T03:04:16.047 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-09-16T03:04:16.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T03:04:16.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T03:04:16.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:04:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:04:16.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:16.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:16.259+0000 7fe5710f17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T03:04:16.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:16.259+0000 7fe5710f17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T03:04:16.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:16.259+0000 7fe5710f17c0 -1 bdev(0x5619b377f800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T03:04:16.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:16.259+0000 7fe5710f17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T03:04:19.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T03:04:19.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:19.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:19.491+0000 7f80b19e37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T03:04:19.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:19.491+0000 7f80b19e37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T03:04:19.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:19.491+0000 7f80b19e37c0 -1 bdev(0x55a0701db800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T03:04:19.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:19.491+0000 7f80b19e37c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T03:04:22.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T03:04:22.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:22.795+0000 7f9d189957c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T03:04:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:22.795+0000 7f9d189957c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T03:04:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:22.795+0000 7f9d189957c0 -1 bdev(0x558303f0f800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T03:04:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:22.795+0000 7f9d189957c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T03:04:25.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T03:04:26.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T03:04:26.046 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T03:04:26.046 INFO:tasks.ceph:role: osd.3 2024-09-16T03:04:26.046 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T03:04:26.046 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:04:26.105 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:04:26.105 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:04:26.105 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:04:26.105 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:04:26.106 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:04:26.106 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:04:26.106 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:04:26.106 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:04:26.106 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:04:26.106 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:04:26.126 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-09-16T03:04:26.127 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-09-16T03:04:26.127 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T03:04:26.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T03:04:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:04:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:04:26.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T03:04:26.302 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T03:04:26.302 INFO:tasks.ceph:role: osd.4 2024-09-16T03:04:26.302 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T03:04:26.302 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:04:26.360 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:04:26.361 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:04:26.361 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:04:26.361 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:04:26.361 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:04:26.361 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:04:26.361 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:04:26.361 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:04:26.361 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:04:26.361 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:04:26.384 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-09-16T03:04:26.386 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-09-16T03:04:26.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T03:04:26.447 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T03:04:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:04:26.499 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:04:26.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T03:04:26.552 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T03:04:26.552 INFO:tasks.ceph:role: osd.5 2024-09-16T03:04:26.552 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T03:04:26.552 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:04:26.605 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:04:26.605 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:04:26.605 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:04:26.605 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:04:26.605 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:04:26.605 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:04:26.605 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:04:26.605 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:04:26.605 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:04:26.607 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:04:26.629 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-09-16T03:04:26.630 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-09-16T03:04:26.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T03:04:26.688 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T03:04:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:04:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:04:26.739 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:26.844+0000 7f5a91dc87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T03:04:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:26.848+0000 7f5a91dc87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T03:04:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:26.848+0000 7f5a91dc87c0 -1 bdev(0x5556a1595800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T03:04:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:26.848+0000 7f5a91dc87c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T03:04:29.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T03:04:30.006 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:30.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:30.088+0000 7f6287a5b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T03:04:30.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:30.088+0000 7f6287a5b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T03:04:30.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:30.088+0000 7f6287a5b7c0 -1 bdev(0x55cd71363800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T03:04:30.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:30.088+0000 7f6287a5b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T03:04:33.202 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T03:04:33.259 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:33.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:33.340+0000 7f384965a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T03:04:33.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:33.340+0000 7f384965a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T03:04:33.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:33.340+0000 7f384965a7c0 -1 bdev(0x55ff1492f800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T03:04:33.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:33.340+0000 7f384965a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T03:04:36.502 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T03:04:36.559 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T03:04:36.559 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:36.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T03:04:36.574 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:36.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T03:04:36.629 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:36.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T03:04:36.685 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:36.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T03:04:36.699 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:36.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T03:04:36.754 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:36.754 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T03:04:36.810 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:36.810 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T03:04:36.865 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:36.865 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T03:04:36.914 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T03:04:36.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:04:36.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:04:36.960 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2024-09-16T03:04:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAwoOdmzSGuDxAAcC1Q76pz6dZvhsiza0RR8Q== 2024-09-16T03:04:36.960 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2024-09-16T03:04:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAzoOdmsOhXHRAA91CSHiUTx4BBReBftiewKw== 2024-09-16T03:04:36.960 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2024-09-16T03:04:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQA2oOdmISeMLxAATRDPsCqPseSW964Yz347YA== 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.x] 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAvoOdmVEQHGBAAJfj2qpZ6xpPtAEZXogxMWA== 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.3] 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQA6oOdmDdmSMhAA0h9skEHjPpEhwtqTOnlRxA== 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.4] 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQA+oOdmOXVTBRAAgvQR6Glo5wNYLFukbQ2WUw== 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.5] 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBBoOdmB3VjFBAA6r3ovgRwqL/B091PVnoheg== 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-09-16T03:04:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAvoOdm5A3nGhAATecQNgTwo9gFLNUAiqqYHQ== 2024-09-16T03:04:36.968 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAwoOdmzSGuDxAAcC1Q76pz6dZvhsiza0RR8Q== 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAzoOdmsOhXHRAA91CSHiUTx4BBReBftiewKw== 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2oOdmISeMLxAATRDPsCqPseSW964Yz347YA== 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAvoOdmVEQHGBAAJfj2qpZ6xpPtAEZXogxMWA== 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.3] 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA6oOdmDdmSMhAA0h9skEHjPpEhwtqTOnlRxA== 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.4] 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA+oOdmOXVTBRAAgvQR6Glo5wNYLFukbQ2WUw== 2024-09-16T03:04:36.969 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.5] 2024-09-16T03:04:36.970 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBBoOdmB3VjFBAA6r3ovgRwqL/B091PVnoheg== 2024-09-16T03:04:36.970 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-09-16T03:04:36.970 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAvoOdm5A3nGhAATecQNgTwo9gFLNUAiqqYHQ== 2024-09-16T03:04:36.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.006 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:04:37.234 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:04:37.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.281 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:04:37.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:04:37.542 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:04:37.624 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T03:04:37.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T03:04:37.641 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:04:37.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T03:04:37.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T03:04:37.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:04:37.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T03:04:37.907 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T03:04:37.920 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:04:38.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T03:04:38.027 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:38.029 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:04:38.074 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T03:04:38.074 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:04:38.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:04:38.078 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:04:38.078 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:04:38.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:04:38.080 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:04:38.081 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:04:38.081 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:04:38.119 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:04:38.119 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T03:04:38.119 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T03:04:38.119 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T03:04:38.122 INFO:tasks.ceph.mgr.x:Started 2024-09-16T03:04:38.122 DEBUG:tasks.ceph:set 0 configs 2024-09-16T03:04:38.122 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T03:04:38.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T03:04:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient: get_monmap_and_config 2024-09-16T03:04:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient: build_initial_monmap 2024-09-16T03:04:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient: monmap: 2024-09-16T03:04:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:38.241786+0000 2024-09-16T03:04:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:38.241786+0000 2024-09-16T03:04:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:38.243 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 1 Processor -- start 2024-09-16T03:04:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 1 -- start start 2024-09-16T03:04:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient: init 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient(hunting): picked mon.noname-a con 0x7eff34135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient(hunting): picked mon.noname-b con 0x7eff3409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient(hunting): picked mon.noname-c con 0x7eff3413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3409f7e0 con 0x7eff34135950 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3409b160 con 0x7eff3413c010 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3409b440 con 0x7eff3409fc40 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3b3d0640 10 monclient(hunting): authenticate will time out at 968.890938s 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3896b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7eff3409fc40 0x7eff3413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58886/0 (socket says 172.21.15.45:58886) 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3916c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7eff34135950 0x7eff34135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:32828/0 (socket says 172.21.15.45:32828) 2024-09-16T03:04:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.239+0000 7eff3896b640 1 -- 172.21.15.45:0/1090428222 learned_addr learned my addr 172.21.15.45:0/1090428222 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:38.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3764849679 0 0) 0x7eff3409f7e0 con 0x7eff34135950 2024-09-16T03:04:38.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient(hunting): _negotiate 2024-09-16T03:04:38.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:38.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:38.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient(hunting): my global_id is 4098 2024-09-16T03:04:38.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff20003530 con 0x7eff34135950 2024-09-16T03:04:38.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041324301 0 0) 0x7eff3409b160 con 0x7eff3413c010 2024-09-16T03:04:38.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient(hunting): _negotiate 2024-09-16T03:04:38.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:38.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:38.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient(hunting): my global_id is 4102 2024-09-16T03:04:38.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff3409f7e0 con 0x7eff3413c010 2024-09-16T03:04:38.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 847571233 0 0) 0x7eff3409f7e0 con 0x7eff3413c010 2024-09-16T03:04:38.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 >> v1:172.21.15.45:6790/0 conn(0x7eff3409fc40 legacy=0x7eff3413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 >> v1:172.21.15.45:6789/0 conn(0x7eff34135950 legacy=0x7eff34135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: _finish_hunting 0 2024-09-16T03:04:38.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 monclient: found mon.noname-c 2024-09-16T03:04:38.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:04:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff34136440 con 0x7eff3413c010 2024-09-16T03:04:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: _finish_auth 0 2024-09-16T03:04:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: _check_auth_tickets 2024-09-16T03:04:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:38.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff24001680 con 0x7eff3413c010 2024-09-16T03:04:38.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:38.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:38.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: dump: 2024-09-16T03:04:38.460 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:38.460 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:38.462 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff3b3d0640 5 monclient: authenticate success, global_id 4102 2024-09-16T03:04:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff3b3d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff24001820 con 0x7eff3413c010 2024-09-16T03:04:38.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:38.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 1 -- 172.21.15.45:0/1090428222 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff24001c70 con 0x7eff3413c010 2024-09-16T03:04:38.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:38.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:38.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff33fff640 10 monclient: dump: 2024-09-16T03:04:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:38.465 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff3b3d0640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.447+0000 7eff3b3d0640 4 set_mon_vals no callback set 2024-09-16T03:04:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.451+0000 7eff3b3d0640 10 monclient: shutdown 2024-09-16T03:04:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.451+0000 7eff3b3d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.451+0000 7eff3b3d0640 1 -- 172.21.15.45:0/1090428222 >> v1:172.21.15.46:6789/0 conn(0x7eff3413c010 legacy=0x7eff3413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.451+0000 7eff3b3d0640 1 -- 172.21.15.45:0/1090428222 shutdown_connections 2024-09-16T03:04:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.451+0000 7eff3b3d0640 1 -- 172.21.15.45:0/1090428222 >> 172.21.15.45:0/1090428222 conn(0x7eff340a2bd0 msgr2=0x7eff340a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:38.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.451+0000 7eff3b3d0640 1 -- 172.21.15.45:0/1090428222 shutdown_connections 2024-09-16T03:04:38.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.451+0000 7eff3b3d0640 1 -- 172.21.15.45:0/1090428222 wait complete. 2024-09-16T03:04:38.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.451+0000 7eff3b3d0640 10 monclient: build_initial_monmap 2024-09-16T03:04:38.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.451+0000 7eff3b3d0640 10 monclient: monmap: 2024-09-16T03:04:38.475 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:04:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:38.477 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:38.477 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:38.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 1 librados: starting msgr at 2024-09-16T03:04:38.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 1 librados: starting objecter 2024-09-16T03:04:38.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 1 Processor -- start 2024-09-16T03:04:38.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 1 -- start start 2024-09-16T03:04:38.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 1 librados: setting wanted keys 2024-09-16T03:04:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 1 librados: calling monclient init 2024-09-16T03:04:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 10 monclient: init 2024-09-16T03:04:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:04:38.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 10 monclient(hunting): picked mon.noname-a con 0x7eff3409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:38.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 10 monclient(hunting): picked mon.noname-b con 0x7eff34135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:38.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 10 monclient(hunting): picked mon.noname-c con 0x7eff3413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:38.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3413ab20 con 0x7eff3409fc40 2024-09-16T03:04:38.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3413f420 con 0x7eff34135950 2024-09-16T03:04:38.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3409d2c0 con 0x7eff3413c010 2024-09-16T03:04:38.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:38.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3b3d0640 10 monclient(hunting): authenticate will time out at 969.106344s 2024-09-16T03:04:38.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3996d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7eff3413c010 0x7eff341d54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58888/0 (socket says 172.21.15.45:58888) 2024-09-16T03:04:38.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff3996d640 1 -- 172.21.15.45:0/2932752051 learned_addr learned my addr 172.21.15.45:0/2932752051 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:38.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 client.?.objecter ms_handle_connect 0x7eff3413c010 2024-09-16T03:04:38.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:38.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 client.?.objecter ms_handle_connect 0x7eff3409fc40 2024-09-16T03:04:38.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:38.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2389750277 0 0) 0x7eff3409d2c0 con 0x7eff3413c010 2024-09-16T03:04:38.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 monclient(hunting): my global_id is 4106 2024-09-16T03:04:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff14004ef0 con 0x7eff3413c010 2024-09-16T03:04:38.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2880231918 0 0) 0x7eff14004ef0 con 0x7eff3413c010 2024-09-16T03:04:38.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff3409d2c0 con 0x7eff3413c010 2024-09-16T03:04:38.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff24001c30 con 0x7eff3413c010 2024-09-16T03:04:38.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:38.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:38.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 monclient(hunting): dump: 2024-09-16T03:04:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:38.486 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:38.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:38.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.455+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1669022081 0 0) 0x7eff3409d2c0 con 0x7eff3413c010 2024-09-16T03:04:38.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 >> v1:172.21.15.46:6789/0 conn(0x7eff34135950 legacy=0x7eff341d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 >> v1:172.21.15.45:6789/0 conn(0x7eff3409fc40 legacy=0x7eff341342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:04:38.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 1 monclient: found mon.c 2024-09-16T03:04:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff340737a0 con 0x7eff3413c010 2024-09-16T03:04:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 10 monclient: _finish_auth 0 2024-09-16T03:04:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:04:38.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 5 monclient: authenticate success, global_id 4106 2024-09-16T03:04:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 10 monclient: _renew_subs 2024-09-16T03:04:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 1 -- 172.21.15.45:0/2932752051 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff34072840 con 0x7eff3413c010 2024-09-16T03:04:38.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 10 client.4106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 10 monclient: _renew_subs 2024-09-16T03:04:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 1 -- 172.21.15.45:0/2932752051 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7eff341349c0 con 0x7eff3413c010 2024-09-16T03:04:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 1 librados: init done 2024-09-16T03:04:38.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff240017c0 con 0x7eff3413c010 2024-09-16T03:04:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff24003b10 con 0x7eff3413c010 2024-09-16T03:04:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff337fe640 4 set_mon_vals no callback set 2024-09-16T03:04:38.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 10 monclient: dump: 2024-09-16T03:04:38.492 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:38.492 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.492 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.492 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.492 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:38.493 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:38.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7eff24004100 con 0x7eff3413c010 2024-09-16T03:04:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2640796692 0 0) 0x7eff24006900 con 0x7eff3413c010 2024-09-16T03:04:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 10 client.4106.objecter ms_dispatch 0x7eff34000ba0 osd_map(1..1 src has 1..1) 2024-09-16T03:04:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 3 client.4106.objecter handle_osd_map got epochs [1,1] > 0 2024-09-16T03:04:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 3 client.4106.objecter handle_osd_map decoding full epoch 1 2024-09-16T03:04:38.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 20 client.4106.objecter dump_active .. 0 homeless 2024-09-16T03:04:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff3b3d0640 1 -- 172.21.15.45:0/2932752051 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff18005090 con 0x7eff3413c010 2024-09-16T03:04:38.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.459+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7eff24006b00 con 0x7eff3413c010 2024-09-16T03:04:38.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.463+0000 7eff31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:38.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.463+0000 7eff31ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:38.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.531+0000 7eff3b3d0640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T03:04:38.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.531+0000 7eff3b3d0640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T03:04:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.531+0000 7eff3b3d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.531+0000 7eff3b3d0640 1 -- 172.21.15.45:0/2932752051 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7eff18005380 con 0x7eff3413c010 2024-09-16T03:04:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.535+0000 7eff31ffb640 1 -- 172.21.15.45:0/2932752051 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7eff24005930 con 0x7eff3413c010 2024-09-16T03:04:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.535+0000 7eff31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T03:04:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.535+0000 7eff31ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T03:04:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2024-09-16T03:04:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 10 librados: watch_flush enter 2024-09-16T03:04:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 10 librados: watch_flush exit 2024-09-16T03:04:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 20 client.4106.objecter shutdown clearing up homeless session... 2024-09-16T03:04:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 10 client.4106.objecter successfully canceled tick 2024-09-16T03:04:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 10 monclient: shutdown 2024-09-16T03:04:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 1 -- 172.21.15.45:0/2932752051 >> v1:172.21.15.45:6790/0 conn(0x7eff3413c010 legacy=0x7eff341d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 1 -- 172.21.15.45:0/2932752051 shutdown_connections 2024-09-16T03:04:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 1 -- 172.21.15.45:0/2932752051 >> 172.21.15.45:0/2932752051 conn(0x7eff340a2bd0 msgr2=0x7eff340a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 1 -- 172.21.15.45:0/2932752051 shutdown_connections 2024-09-16T03:04:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 1 -- 172.21.15.45:0/2932752051 wait complete. 2024-09-16T03:04:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:38.539+0000 7eff3b3d0640 1 librados: shutdown 2024-09-16T03:04:38.559 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T03:04:38.559 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T03:04:38.559 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:38.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T03:04:38.570 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:38.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T03:04:38.623 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-16T03:04:38.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T03:04:38.674 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:38.674 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T03:04:38.687 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:38.687 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T03:04:38.742 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:04:38.742 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T03:04:38.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new 60a0a607-fe6c-4bd5-b369-e0dd6f6bd551 0 2024-09-16T03:04:38.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.960+0000 7f8e2aa8f640 10 monclient: get_monmap_and_config 2024-09-16T03:04:38.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.960+0000 7f8e2aa8f640 10 monclient: build_initial_monmap 2024-09-16T03:04:38.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.960+0000 7f8e2aa8f640 10 monclient: monmap: 2024-09-16T03:04:38.964 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:38.964 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.964 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:38.962846+0000 2024-09-16T03:04:38.964 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:38.962846+0000 2024-09-16T03:04:38.964 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:38.965 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:38.965 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:38.965 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:38.965 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:38.965 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:38.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.960+0000 7f8e2aa8f640 1 Processor -- start 2024-09-16T03:04:38.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.960+0000 7f8e2aa8f640 1 -- start start 2024-09-16T03:04:38.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.960+0000 7f8e2aa8f640 10 monclient: init 2024-09-16T03:04:38.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): picked mon.noname-c con 0x7f8e24135860 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): picked mon.noname-a con 0x7f8e241388c0 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): picked mon.noname-b con 0x7f8e2413c3d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e24076b90 con 0x7f8e241388c0 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e2409cb10 con 0x7f8e24135860 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e240a0650 con 0x7f8e2413c3d0 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): authenticate will time out at 962.684306s 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2882b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8e24135860 0x7f8e24135c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58184/0 (socket says 172.21.15.46:58184) 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2882b640 1 -- 172.21.15.46:0/3306027103 learned_addr learned my addr 172.21.15.46:0/3306027103 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 -- 172.21.15.46:0/3306027103 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1545865072 0 0) 0x7f8e14001120 con 0x7f8e24135860 2024-09-16T03:04:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient(hunting): my global_id is 4105 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 -- 172.21.15.46:0/3306027103 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e10003530 con 0x7f8e24135860 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 -- 172.21.15.46:0/3306027103 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 498694984 0 0) 0x7f8e10003530 con 0x7f8e24135860 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 -- 172.21.15.46:0/3306027103 >> v1:172.21.15.45:6790/0 conn(0x7f8e2413c3d0 legacy=0x7f8e2413e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 -- 172.21.15.46:0/3306027103 >> v1:172.21.15.45:6789/0 conn(0x7f8e241388c0 legacy=0x7f8e2413acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: _finish_hunting 0 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 monclient: found mon.noname-c 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 -- 172.21.15.46:0/3306027103 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2409fdb0 con 0x7f8e24135860 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: _finish_auth 0 2024-09-16T03:04:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: _check_auth_tickets 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 -- 172.21.15.46:0/3306027103 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e14001870 con 0x7f8e24135860 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 5 monclient: authenticate success, global_id 4105 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: dump: 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 -- 172.21.15.46:0/3306027103 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e14001a90 con 0x7f8e24135860 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 4 set_mon_vals no callback set 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 1 -- 172.21.15.46:0/3306027103 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e140048b0 con 0x7f8e24135860 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e237fe640 10 monclient: dump: 2024-09-16T03:04:38.972 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:38.974 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient: shutdown 2024-09-16T03:04:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/3306027103 >> v1:172.21.15.46:6789/0 conn(0x7f8e24135860 legacy=0x7f8e24135c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/3306027103 shutdown_connections 2024-09-16T03:04:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/3306027103 >> 172.21.15.46:0/3306027103 conn(0x7f8e240a3350 msgr2=0x7f8e240a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/3306027103 shutdown_connections 2024-09-16T03:04:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/3306027103 wait complete. 2024-09-16T03:04:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient: build_initial_monmap 2024-09-16T03:04:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient: monmap: 2024-09-16T03:04:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T03:04:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T03:04:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:38.976 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:38.976 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:38.976 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:38.976 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:38.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 librados: starting msgr at 2024-09-16T03:04:38.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 librados: starting objecter 2024-09-16T03:04:38.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 Processor -- start 2024-09-16T03:04:38.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- start start 2024-09-16T03:04:38.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 librados: setting wanted keys 2024-09-16T03:04:38.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 librados: calling monclient init 2024-09-16T03:04:38.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient: init 2024-09-16T03:04:38.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:38.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:04:38.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): picked mon.noname-b con 0x7f8e24135860 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:38.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): picked mon.noname-a con 0x7f8e241388c0 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:38.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): picked mon.noname-c con 0x7f8e2413c3d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e240a5a40 con 0x7f8e241388c0 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e2411d3d0 con 0x7f8e24135860 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e2411d6b0 con 0x7f8e2413c3d0 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2aa8f640 10 monclient(hunting): authenticate will time out at 962.687502s 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2882b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8e24135860 0x7f8e24131be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58200/0 (socket says 172.21.15.46:58200) 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e2882b640 1 -- 172.21.15.46:0/1079347914 learned_addr learned my addr 172.21.15.46:0/1079347914 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 10 client.?.objecter ms_handle_connect 0x7f8e24135860 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2447589740 0 0) 0x7f8e2411d3d0 con 0x7f8e24135860 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 10 monclient(hunting): _negotiate 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:38.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:38.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 10 monclient(hunting): my global_id is 4108 2024-09-16T03:04:38.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e04004ec0 con 0x7f8e24135860 2024-09-16T03:04:38.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 10 client.?.objecter ms_handle_connect 0x7f8e241388c0 2024-09-16T03:04:38.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.964+0000 7f8e217fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:38.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 client.?.objecter ms_handle_connect 0x7f8e2413c3d0 2024-09-16T03:04:38.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:38.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 966847542 0 0) 0x7f8e24076b90 con 0x7f8e241388c0 2024-09-16T03:04:38.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): _negotiate 2024-09-16T03:04:38.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): my global_id is 4107 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e2411d3d0 con 0x7f8e241388c0 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3204009964 0 0) 0x7f8e04004ec0 con 0x7f8e24135860 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e24076b90 con 0x7f8e24135860 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e14000ff0 con 0x7f8e24135860 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): dump: 2024-09-16T03:04:38.984 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 17411250 0 0) 0x7f8e2411d6b0 con 0x7f8e2413c3d0 2024-09-16T03:04:38.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): _negotiate 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient(hunting): my global_id is 4115 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e04004ec0 con 0x7f8e2413c3d0 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2849053611 0 0) 0x7f8e24076b90 con 0x7f8e24135860 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 >> v1:172.21.15.45:6790/0 conn(0x7f8e2413c3d0 legacy=0x7f8e241d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 >> v1:172.21.15.45:6789/0 conn(0x7f8e241388c0 legacy=0x7f8e241cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: _finish_hunting 0 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 monclient: found mon.b 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:38.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e240731f0 con 0x7f8e24135860 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: _finish_auth 0 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: _check_auth_tickets 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 5 monclient: authenticate success, global_id 4108 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 10 monclient: _renew_subs 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/1079347914 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e24071be0 con 0x7f8e24135860 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 10 monclient: _renew_subs 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/1079347914 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e241324c0 con 0x7f8e24135860 2024-09-16T03:04:38.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 1 librados: init done 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e140019f0 con 0x7f8e24135860 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e14004d10 con 0x7f8e24135860 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e22ffd640 4 set_mon_vals no callback set 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: dump: 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.988 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8e14005240 con 0x7f8e24135860 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 4240644994 0 0) 0x7f8e14006630 con 0x7f8e24135860 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 client.4108.objecter ms_dispatch 0x7f8e24000ba0 osd_map(2..2 src has 1..2) 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 3 client.4108.objecter handle_osd_map got epochs [2,2] > 0 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 3 client.4108.objecter handle_osd_map decoding full epoch 2 2024-09-16T03:04:38.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 20 client.4108.objecter dump_active .. 0 homeless 2024-09-16T03:04:38.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:38.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:38.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:38.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/1079347914 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e08005090 con 0x7f8e24135860 2024-09-16T03:04:38.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f8e2409cb10 con 0x7f8e24135860 2024-09-16T03:04:38.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:38.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:38.968+0000 7f8e217fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:39.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.040+0000 7f8e2aa8f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "60a0a607-fe6c-4bd5-b369-e0dd6f6bd551", "id": 0}] 2024-09-16T03:04:39.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.040+0000 7f8e2aa8f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "60a0a607-fe6c-4bd5-b369-e0dd6f6bd551", "id": 0}] 2024-09-16T03:04:39.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.040+0000 7f8e2aa8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.040+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/1079347914 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd new", "uuid": "60a0a607-fe6c-4bd5-b369-e0dd6f6bd551", "id": 0} v 0) -- 0x7f8e08005380 con 0x7f8e24135860 2024-09-16T03:04:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e217fa640 1 -- 172.21.15.46:0/1079347914 <== mon.1 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "60a0a607-fe6c-4bd5-b369-e0dd6f6bd551", "id": 0}]=0 v3) ==== 112+0+2 (unknown 3931353509 0 732707515) 0x7f8e14006880 con 0x7f8e24135860 2024-09-16T03:04:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "60a0a607-fe6c-4bd5-b369-e0dd6f6bd551", "id": 0}] 2024-09-16T03:04:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e217fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:39.051 INFO:teuthology.orchestra.run.smithi046.stdout:0 2024-09-16T03:04:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 10 librados: watch_flush enter 2024-09-16T03:04:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 10 librados: watch_flush exit 2024-09-16T03:04:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 20 client.4108.objecter shutdown clearing up homeless session... 2024-09-16T03:04:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 10 client.4108.objecter successfully canceled tick 2024-09-16T03:04:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 10 monclient: shutdown 2024-09-16T03:04:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/1079347914 >> v1:172.21.15.46:6789/0 conn(0x7f8e24135860 legacy=0x7f8e24131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/1079347914 shutdown_connections 2024-09-16T03:04:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/1079347914 >> 172.21.15.46:0/1079347914 conn(0x7f8e240a3350 msgr2=0x7f8e241345a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/1079347914 shutdown_connections 2024-09-16T03:04:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 1 -- 172.21.15.46:0/1079347914 wait complete. 2024-09-16T03:04:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.048+0000 7f8e2aa8f640 1 librados: shutdown 2024-09-16T03:04:39.067 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new 10adca9c-3542-4901-a439-a1c9e9fcb8c1 1 2024-09-16T03:04:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient: get_monmap_and_config 2024-09-16T03:04:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient: build_initial_monmap 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient: monmap: 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:39.182308+0000 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:39.182308+0000 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:39.184 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 1 Processor -- start 2024-09-16T03:04:39.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 1 -- start start 2024-09-16T03:04:39.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient: init 2024-09-16T03:04:39.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:39.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:04:39.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient(hunting): picked mon.noname-c con 0x7fec7009c9f0 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:39.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient(hunting): picked mon.noname-b con 0x7fec700a6330 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient(hunting): picked mon.noname-a con 0x7fec7013de10 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec700a87d0 con 0x7fec7013de10 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec700a4cb0 con 0x7fec7009c9f0 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec700a4f90 con 0x7fec700a6330 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient(hunting): authenticate will time out at 962.903032s 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fec7009c9f0 0x7fec7009cdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58202/0 (socket says 172.21.15.46:58202) 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6e59c640 1 -- 172.21.15.46:0/668823427 learned_addr learned my addr 172.21.15.46:0/668823427 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 -- 172.21.15.46:0/668823427 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1595994355 0 0) 0x7fec700a4cb0 con 0x7fec7009c9f0 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient(hunting): my global_id is 4111 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 -- 172.21.15.46:0/668823427 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec5c003530 con 0x7fec7009c9f0 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 -- 172.21.15.46:0/668823427 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3040599150 0 0) 0x7fec5c003530 con 0x7fec7009c9f0 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 -- 172.21.15.46:0/668823427 >> v1:172.21.15.45:6790/0 conn(0x7fec700a6330 legacy=0x7fec7009d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 -- 172.21.15.46:0/668823427 >> v1:172.21.15.45:6789/0 conn(0x7fec7013de10 legacy=0x7fec70140200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: _finish_hunting 0 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 monclient: found mon.noname-c 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 -- 172.21.15.46:0/668823427 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec700a59f0 con 0x7fec7009c9f0 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: _finish_auth 0 2024-09-16T03:04:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: _check_auth_tickets 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 -- 172.21.15.46:0/668823427 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fec68001a20 con 0x7fec7009c9f0 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 5 monclient: authenticate success, global_id 4111 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: dump: 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 -- 172.21.15.46:0/668823427 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec68001bc0 con 0x7fec7009c9f0 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 1 -- 172.21.15.46:0/668823427 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fec68002010 con 0x7fec7009c9f0 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec74d2e640 4 set_mon_vals no callback set 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.180+0000 7fec6d59a640 10 monclient: dump: 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: shutdown 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- 172.21.15.46:0/668823427 >> v1:172.21.15.46:6789/0 conn(0x7fec7009c9f0 legacy=0x7fec7009cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- 172.21.15.46:0/668823427 shutdown_connections 2024-09-16T03:04:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- 172.21.15.46:0/668823427 >> 172.21.15.46:0/668823427 conn(0x7fec700aa2e0 msgr2=0x7fec700aa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- 172.21.15.46:0/668823427 shutdown_connections 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- 172.21.15.46:0/668823427 wait complete. 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: build_initial_monmap 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: monmap: 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:39.197 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 librados: starting msgr at 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 librados: starting objecter 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 Processor -- start 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- start start 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 librados: setting wanted keys 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 librados: calling monclient init 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: init 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient(hunting): picked mon.noname-a con 0x7fec7009c9f0 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient(hunting): picked mon.noname-b con 0x7fec700a6330 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient(hunting): picked mon.noname-c con 0x7fec7013de10 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec7013c0c0 con 0x7fec7009c9f0 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec7013c3a0 con 0x7fec700a6330 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec70139c00 con 0x7fec7013de10 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient(hunting): authenticate will time out at 962.905956s 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec6dd9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fec700a6330 0x7fec701394f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58208/0 (socket says 172.21.15.46:58208) 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec6dd9b640 1 -- 172.21.15.46:0/3208057239 learned_addr learned my addr 172.21.15.46:0/3208057239 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 client.?.objecter ms_handle_connect 0x7fec700a6330 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023084688 0 0) 0x7fec7013c3a0 con 0x7fec700a6330 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): my global_id is 4114 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec50004ec0 con 0x7fec700a6330 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 client.?.objecter ms_handle_connect 0x7fec7009c9f0 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 client.?.objecter ms_handle_connect 0x7fec7013de10 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1167287444 0 0) 0x7fec50004ec0 con 0x7fec700a6330 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec7013c3a0 con 0x7fec700a6330 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fec58001a10 con 0x7fec700a6330 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): dump: 2024-09-16T03:04:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2209120996 0 0) 0x7fec7013c0c0 con 0x7fec7009c9f0 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): my global_id is 4113 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec50004ec0 con 0x7fec7009c9f0 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2829823450 0 0) 0x7fec70139c00 con 0x7fec7013de10 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient(hunting): my global_id is 4121 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec7013c0c0 con 0x7fec7013de10 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4034398424 0 0) 0x7fec7013c3a0 con 0x7fec700a6330 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 >> v1:172.21.15.45:6790/0 conn(0x7fec7013de10 legacy=0x7fec701d4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 >> v1:172.21.15.45:6789/0 conn(0x7fec7009c9f0 legacy=0x7fec70138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 monclient: found mon.b 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec700a1300 con 0x7fec700a6330 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient: _finish_auth 0 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 5 monclient: authenticate success, global_id 4114 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: _renew_subs 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec7013b530 con 0x7fec700a6330 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: _renew_subs 2024-09-16T03:04:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fec701386b0 con 0x7fec700a6330 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec74d2e640 1 librados: init done 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec580038d0 con 0x7fec700a6330 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec67fff640 4 set_mon_vals no callback set 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fec58003b00 con 0x7fec700a6330 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 monclient: dump: 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fec580040f0 con 0x7fec700a6330 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 1598605611 0 0) 0x7fec580066d0 con 0x7fec700a6330 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 10 client.4114.objecter ms_dispatch 0x7fec70000ba0 osd_map(3..3 src has 1..3) 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 3 client.4114.objecter handle_osd_map got epochs [3,3] > 0 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 3 client.4114.objecter handle_osd_map decoding full epoch 3 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec66ffd640 20 client.4114.objecter dump_active .. 0 homeless 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec64ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec64ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec64ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.184+0000 7fec64ff9640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec48005090 con 0x7fec700a6330 2024-09-16T03:04:39.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.188+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fec58006920 con 0x7fec700a6330 2024-09-16T03:04:39.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.188+0000 7fec66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.188+0000 7fec66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec64ff9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "10adca9c-3542-4901-a439-a1c9e9fcb8c1", "id": 1}] 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec64ff9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "10adca9c-3542-4901-a439-a1c9e9fcb8c1", "id": 1}] 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec64ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec64ff9640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd new", "uuid": "10adca9c-3542-4901-a439-a1c9e9fcb8c1", "id": 1} v 0) -- 0x7fec48005380 con 0x7fec700a6330 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec6dd9b640 1 -- 172.21.15.46:0/3208057239 >> v1:172.21.15.46:6789/0 conn(0x7fec700a6330 legacy=0x7fec701394f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec6dd9b640 1 -- 172.21.15.46:0/3208057239 >> v1:172.21.15.46:6789/0 conn(0x7fec700a6330 legacy=0x7fec701394f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec6dd9b640 1 --1- 172.21.15.46:0/3208057239 >> v1:172.21.15.46:6789/0 conn(0x7fec700a6330 0x7fec701394f0 :-1 s=OPENED pgs=16 cs=1 l=1).handle_message read tag failed 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec6dd9b640 1 --1- 172.21.15.46:0/3208057239 >> v1:172.21.15.46:6789/0 conn(0x7fec700a6330 0x7fec701394f0 :-1 s=OPENED pgs=16 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.46:6789/0 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 >> v1:172.21.15.46:6789/0 conn(0x7fec700a6330 legacy=0x7fec701394f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:04:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient(hunting): picked mon.c con 0x7fec7013de10 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient(hunting): picked mon.a con 0x7fec7009c9f0 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient(hunting): picked mon.b con 0x7fec5000faa0 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fec7013c3a0 con 0x7fec7009c9f0 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fec70139c00 con 0x7fec5000faa0 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fec50015ba0 con 0x7fec7013de10 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 client.4114.objecter ms_handle_connect 0x7fec5000faa0 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 client.4114.objecter resend_mon_ops 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 client.4114.objecter ms_handle_connect 0x7fec7013de10 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 client.4114.objecter resend_mon_ops 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129991774 0 0) 0x7fec70139c00 con 0x7fec5000faa0 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fec50015d70 con 0x7fec5000faa0 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 client.4114.objecter ms_handle_connect 0x7fec7009c9f0 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 client.4114.objecter resend_mon_ops 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 1572594130 0 0) 0x7fec50015d70 con 0x7fec5000faa0 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 >> v1:172.21.15.45:6790/0 conn(0x7fec7013de10 legacy=0x7fec5000a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 >> v1:172.21.15.45:6789/0 conn(0x7fec7009c9f0 legacy=0x7fec5000e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 monclient: found mon.b 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7fec50016e40 con 0x7fec5000faa0 2024-09-16T03:04:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "10adca9c-3542-4901-a439-a1c9e9fcb8c1", "id": 1}] 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd new", "uuid": "10adca9c-3542-4901-a439-a1c9e9fcb8c1", "id": 1} v 0) -- 0x7fec70139c00 con 0x7fec5000faa0 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: _finish_auth 0 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fec600015c0 con 0x7fec5000faa0 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: dump: 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec60001760 con 0x7fec5000faa0 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec67fff640 4 set_mon_vals no callback set 2024-09-16T03:04:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.260+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fec60003600 con 0x7fec5000faa0 2024-09-16T03:04:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec66ffd640 1 -- 172.21.15.46:0/3208057239 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_command_ack([{"prefix": "osd new", "uuid": "10adca9c-3542-4901-a439-a1c9e9fcb8c1", "id": 1}]=0 v4) ==== 112+0+2 (unknown 2007870181 0 940483276) 0x7fec60004860 con 0x7fec5000faa0 2024-09-16T03:04:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "10adca9c-3542-4901-a439-a1c9e9fcb8c1", "id": 1}] 2024-09-16T03:04:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:39.270 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-09-16T03:04:39.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec64ff9640 10 librados: watch_flush enter 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec64ff9640 10 librados: watch_flush exit 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec64ff9640 20 client.4114.objecter shutdown clearing up homeless session... 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec64ff9640 10 client.4114.objecter successfully canceled tick 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec64ff9640 10 monclient: shutdown 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec64ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.264+0000 7fec64ff9640 1 -- 172.21.15.46:0/3208057239 >> v1:172.21.15.46:6789/0 conn(0x7fec5000faa0 legacy=0x7fec50011f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.268+0000 7fec64ff9640 1 -- 172.21.15.46:0/3208057239 shutdown_connections 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.268+0000 7fec64ff9640 1 -- 172.21.15.46:0/3208057239 >> 172.21.15.46:0/3208057239 conn(0x7fec700aa2e0 msgr2=0x7fec7013d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.268+0000 7fec64ff9640 1 -- 172.21.15.46:0/3208057239 shutdown_connections 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.268+0000 7fec64ff9640 1 -- 172.21.15.46:0/3208057239 wait complete. 2024-09-16T03:04:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.268+0000 7fec64ff9640 1 librados: shutdown 2024-09-16T03:04:39.284 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new 88fe1895-098a-4f13-95b2-4ead1739e71c 2 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient: get_monmap_and_config 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient: build_initial_monmap 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient: monmap: 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:39.405956+0000 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:39.405956+0000 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:39.408 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 1 Processor -- start 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 1 -- start start 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient: init 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient(hunting): picked mon.noname-c con 0x7f6478068960 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient(hunting): picked mon.noname-a con 0x7f6478069450 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient(hunting): picked mon.noname-b con 0x7f647809dc60 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64780a2500 con 0x7f6478069450 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64780a21b0 con 0x7f6478068960 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64780a2e90 con 0x7f647809dc60 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient(hunting): authenticate will time out at 963.126879s 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6476d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6478068960 0x7f6478068d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58228/0 (socket says 172.21.15.46:58228) 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6476d9d640 1 -- 172.21.15.46:0/2498863283 learned_addr learned my addr 172.21.15.46:0/2498863283 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 -- 172.21.15.46:0/2498863283 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700307686 0 0) 0x7f64780a21b0 con 0x7f6478068960 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient(hunting): my global_id is 4117 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 -- 172.21.15.46:0/2498863283 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6460003530 con 0x7f6478068960 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 -- 172.21.15.46:0/2498863283 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 165150075 0 0) 0x7f6460003530 con 0x7f6478068960 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 -- 172.21.15.46:0/2498863283 >> v1:172.21.15.45:6790/0 conn(0x7f647809dc60 legacy=0x7f647813e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 -- 172.21.15.46:0/2498863283 >> v1:172.21.15.45:6789/0 conn(0x7f6478069450 legacy=0x7f647813ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:04:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 monclient: found mon.noname-c 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 -- 172.21.15.46:0/2498863283 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64780a3060 con 0x7f6478068960 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: _finish_auth 0 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 5 monclient: authenticate success, global_id 4117 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 -- 172.21.15.46:0/2498863283 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f64680016a0 con 0x7f6478068960 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: dump: 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 -- 172.21.15.46:0/2498863283 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6468001840 con 0x7f6478068960 2024-09-16T03:04:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 1 -- 172.21.15.46:0/2498863283 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6468001c90 con 0x7f6478068960 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f647d0e9640 4 set_mon_vals no callback set 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.404+0000 7f6475d9b640 10 monclient: dump: 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: shutdown 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- 172.21.15.46:0/2498863283 >> v1:172.21.15.46:6789/0 conn(0x7f6478068960 legacy=0x7f6478068d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- 172.21.15.46:0/2498863283 shutdown_connections 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- 172.21.15.46:0/2498863283 >> 172.21.15.46:0/2498863283 conn(0x7f64780a3ae0 msgr2=0x7f64780a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- 172.21.15.46:0/2498863283 shutdown_connections 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- 172.21.15.46:0/2498863283 wait complete. 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: build_initial_monmap 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: monmap: 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T03:04:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 librados: starting msgr at 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 librados: starting objecter 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 Processor -- start 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- start start 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 librados: setting wanted keys 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 librados: calling monclient init 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: init 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient(hunting): picked mon.noname-b con 0x7f6478068960 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient(hunting): picked mon.noname-c con 0x7f6478069450 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient(hunting): picked mon.noname-a con 0x7f647809dc60 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f647809e1d0 con 0x7f647809dc60 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6478070060 con 0x7f6478068960 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6478136f00 con 0x7f6478069450 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient(hunting): authenticate will time out at 963.129679s 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f6476d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6478068960 0x7f64781ccfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58244/0 (socket says 172.21.15.46:58244) 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f6476d9d640 1 -- 172.21.15.46:0/1109175907 learned_addr learned my addr 172.21.15.46:0/1109175907 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 client.?.objecter ms_handle_connect 0x7f6478068960 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682506484 0 0) 0x7f6478070060 con 0x7f6478068960 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): my global_id is 4120 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6464004ef0 con 0x7f6478068960 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 client.?.objecter ms_handle_connect 0x7f6478069450 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 client.?.objecter ms_handle_connect 0x7f647809dc60 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1532276716 0 0) 0x7f6464004ef0 con 0x7f6478068960 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6478070060 con 0x7f6478068960 2024-09-16T03:04:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6468001bd0 con 0x7f6478068960 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): dump: 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2084676500 0 0) 0x7f6478136f00 con 0x7f6478069450 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): my global_id is 4127 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6464004ef0 con 0x7f6478069450 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2780094292 0 0) 0x7f647809e1d0 con 0x7f647809dc60 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient(hunting): my global_id is 4119 2024-09-16T03:04:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6478136f00 con 0x7f647809dc60 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1032189683 0 0) 0x7f6478070060 con 0x7f6478068960 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 >> v1:172.21.15.45:6790/0 conn(0x7f6478069450 legacy=0x7f647809f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 >> v1:172.21.15.45:6789/0 conn(0x7f647809dc60 legacy=0x7f64781d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 monclient: found mon.b 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64781379b0 con 0x7f6478068960 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 5 monclient: authenticate success, global_id 4120 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: _renew_subs 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- 172.21.15.46:0/1109175907 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6478136c50 con 0x7f6478068960 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: _renew_subs 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- 172.21.15.46:0/1109175907 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f64781cd8a0 con 0x7f6478068960 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 librados: init done 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64680017a0 con 0x7f6478068960 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6468004da0 con 0x7f6478068960 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647559a640 4 set_mon_vals no callback set 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 monclient: dump: 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6468005350 con 0x7f6478068960 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 659627638 0 0) 0x7f64680065c0 con 0x7f6478068960 2024-09-16T03:04:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 10 client.4120.objecter ms_dispatch 0x7f6478000ba0 osd_map(4..4 src has 1..4) 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 3 client.4120.objecter handle_osd_map got epochs [4,4] > 0 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 3 client.4120.objecter handle_osd_map decoding full epoch 4 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f645f7fe640 20 client.4120.objecter dump_active .. 0 homeless 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.408+0000 7f647d0e9640 1 -- 172.21.15.46:0/1109175907 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6458005090 con 0x7f6478068960 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.412+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f6468006990 con 0x7f6478068960 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.412+0000 7f645f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.412+0000 7f645f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.480+0000 7f647d0e9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "88fe1895-098a-4f13-95b2-4ead1739e71c", "id": 2}] 2024-09-16T03:04:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.480+0000 7f647d0e9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "88fe1895-098a-4f13-95b2-4ead1739e71c", "id": 2}] 2024-09-16T03:04:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.480+0000 7f647d0e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.480+0000 7f647d0e9640 1 -- 172.21.15.46:0/1109175907 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd new", "uuid": "88fe1895-098a-4f13-95b2-4ead1739e71c", "id": 2} v 0) -- 0x7f6458005380 con 0x7f6478068960 2024-09-16T03:04:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f645f7fe640 1 -- 172.21.15.46:0/1109175907 <== mon.1 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "88fe1895-098a-4f13-95b2-4ead1739e71c", "id": 2}]=0 v5) ==== 112+0+2 (unknown 361247438 0 216599125) 0x7f6468005640 con 0x7f6478068960 2024-09-16T03:04:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f645f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "88fe1895-098a-4f13-95b2-4ead1739e71c", "id": 2}] 2024-09-16T03:04:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f645f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:39.491 INFO:teuthology.orchestra.run.smithi046.stdout:2 2024-09-16T03:04:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 10 librados: watch_flush enter 2024-09-16T03:04:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 10 librados: watch_flush exit 2024-09-16T03:04:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 20 client.4120.objecter shutdown clearing up homeless session... 2024-09-16T03:04:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 10 client.4120.objecter successfully canceled tick 2024-09-16T03:04:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 10 monclient: shutdown 2024-09-16T03:04:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 1 -- 172.21.15.46:0/1109175907 >> v1:172.21.15.46:6789/0 conn(0x7f6478068960 legacy=0x7f64781ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 1 -- 172.21.15.46:0/1109175907 shutdown_connections 2024-09-16T03:04:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 1 -- 172.21.15.46:0/1109175907 >> 172.21.15.46:0/1109175907 conn(0x7f64780a3ae0 msgr2=0x7f64780a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 1 -- 172.21.15.46:0/1109175907 shutdown_connections 2024-09-16T03:04:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 1 -- 172.21.15.46:0/1109175907 wait complete. 2024-09-16T03:04:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.488+0000 7f647d0e9640 1 librados: shutdown 2024-09-16T03:04:39.506 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new b04f94bc-5631-478e-904c-1dfc9a673a7e 3 2024-09-16T03:04:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient: get_monmap_and_config 2024-09-16T03:04:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient: build_initial_monmap 2024-09-16T03:04:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient: monmap: 2024-09-16T03:04:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:39.618281+0000 2024-09-16T03:04:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:39.618281+0000 2024-09-16T03:04:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:39.620 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 1 Processor -- start 2024-09-16T03:04:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 1 -- start start 2024-09-16T03:04:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient: init 2024-09-16T03:04:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:04:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient(hunting): picked mon.noname-b con 0x7f88100a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient(hunting): picked mon.noname-a con 0x7f88100a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient(hunting): picked mon.noname-c con 0x7f8810073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881009d9e0 con 0x7f88100a5810 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881009dc20 con 0x7f8810073d40 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8810073360 con 0x7f88100a4d20 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f8815735640 10 monclient(hunting): authenticate will time out at 963.339305s 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f880f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8810073d40 0x7f8810140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58258/0 (socket says 172.21.15.46:58258) 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f880f7fe640 1 -- 172.21.15.46:0/2975331417 learned_addr learned my addr 172.21.15.46:0/2975331417 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f880dffb640 1 -- 172.21.15.46:0/2975331417 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1450307307 0 0) 0x7f881009dc20 con 0x7f8810073d40 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f880dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f880dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f880dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f880dffb640 10 monclient(hunting): my global_id is 4123 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.616+0000 7f880dffb640 1 -- 172.21.15.46:0/2975331417 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f8003530 con 0x7f8810073d40 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 1 -- 172.21.15.46:0/2975331417 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3686594856 0 0) 0x7f87f8003530 con 0x7f8810073d40 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 1 -- 172.21.15.46:0/2975331417 >> v1:172.21.15.45:6790/0 conn(0x7f88100a4d20 legacy=0x7f88100a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 1 -- 172.21.15.46:0/2975331417 >> v1:172.21.15.45:6789/0 conn(0x7f88100a5810 legacy=0x7f881013cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 1 monclient: found mon.noname-c 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 1 -- 172.21.15.46:0/2975331417 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881009cf30 con 0x7f8810073d40 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: _finish_auth 0 2024-09-16T03:04:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 1 -- 172.21.15.46:0/2975331417 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8808001a20 con 0x7f8810073d40 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: dump: 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.630 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 5 monclient: authenticate success, global_id 4123 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 1 -- 172.21.15.46:0/2975331417 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8808001cf0 con 0x7f8810073d40 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 1 -- 172.21.15.46:0/2975331417 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8808004c30 con 0x7f8810073d40 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880dffb640 10 monclient: dump: 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 4 set_mon_vals no callback set 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient: shutdown 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 -- 172.21.15.46:0/2975331417 >> v1:172.21.15.46:6789/0 conn(0x7f8810073d40 legacy=0x7f8810140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 -- 172.21.15.46:0/2975331417 shutdown_connections 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 -- 172.21.15.46:0/2975331417 >> 172.21.15.46:0/2975331417 conn(0x7f88100aa720 msgr2=0x7f88100aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 -- 172.21.15.46:0/2975331417 shutdown_connections 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 -- 172.21.15.46:0/2975331417 wait complete. 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient: build_initial_monmap 2024-09-16T03:04:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient: monmap: 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 librados: starting msgr at 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 librados: starting objecter 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 Processor -- start 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 -- start start 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 librados: setting wanted keys 2024-09-16T03:04:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 librados: calling monclient init 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient: init 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient(hunting): picked mon.noname-a con 0x7f8810073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient(hunting): picked mon.noname-c con 0x7f88100a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient(hunting): picked mon.noname-b con 0x7f88100a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88100a3800 con 0x7f8810073d40 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881013f0d0 con 0x7f88100a5810 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8810140490 con 0x7f88100a4d20 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f8815735640 10 monclient(hunting): authenticate will time out at 963.343421s 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f88100a5810 0x7f88101d4fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58274/0 (socket says 172.21.15.46:58274) 2024-09-16T03:04:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f880f7fe640 1 -- 172.21.15.46:0/632124427 learned_addr learned my addr 172.21.15.46:0/632124427 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 10 client.?.objecter ms_handle_connect 0x7f88100a5810 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 10 client.?.objecter ms_handle_connect 0x7f88100a4d20 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2103068949 0 0) 0x7f881013f0d0 con 0x7f88100a5810 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 10 monclient(hunting): my global_id is 4126 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.620+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87fc004ec0 con 0x7f88100a5810 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2506182126 0 0) 0x7f8810140490 con 0x7f88100a4d20 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient(hunting): my global_id is 4133 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f881013f0d0 con 0x7f88100a4d20 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4268395639 0 0) 0x7f87fc004ec0 con 0x7f88100a5810 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8810140490 con 0x7f88100a5810 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8808001840 con 0x7f88100a5810 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient(hunting): dump: 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.636 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3690101677 0 0) 0x7f8810140490 con 0x7f88100a5810 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 >> v1:172.21.15.45:6790/0 conn(0x7f88100a4d20 legacy=0x7f881013cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 >> v1:172.21.15.45:6789/0 conn(0x7f8810073d40 legacy=0x7f881013c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 monclient: found mon.b 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881013d490 con 0x7f88100a5810 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: _finish_auth 0 2024-09-16T03:04:39.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 5 monclient: authenticate success, global_id 4126 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 10 monclient: _renew_subs 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 1 -- 172.21.15.46:0/632124427 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88100a1510 con 0x7f88100a5810 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 10 monclient: _renew_subs 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 1 -- 172.21.15.46:0/632124427 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8810140c10 con 0x7f88100a5810 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 1 librados: init done 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8808001a60 con 0x7f88100a5810 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8808005380 con 0x7f88100a5810 2024-09-16T03:04:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: dump: 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f88080058b0 con 0x7f88100a5810 2024-09-16T03:04:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f880d7fa640 4 set_mon_vals no callback set 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 1807362689 0 0) 0x7f8808006030 con 0x7f88100a5810 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 client.4126.objecter ms_dispatch 0x7f8810000ba0 osd_map(5..5 src has 1..5) 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 3 client.4126.objecter handle_osd_map got epochs [5,5] > 0 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 3 client.4126.objecter handle_osd_map decoding full epoch 5 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 20 client.4126.objecter dump_active .. 0 homeless 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f8815735640 1 -- 172.21.15.46:0/632124427 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f0005090 con 0x7f88100a5810 2024-09-16T03:04:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f8808005ba0 con 0x7f88100a5810 2024-09-16T03:04:39.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.624+0000 7f87f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:39.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.696+0000 7f8815735640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b04f94bc-5631-478e-904c-1dfc9a673a7e", "id": 3}] 2024-09-16T03:04:39.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.696+0000 7f8815735640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b04f94bc-5631-478e-904c-1dfc9a673a7e", "id": 3}] 2024-09-16T03:04:39.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.696+0000 7f8815735640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.696+0000 7f8815735640 1 -- 172.21.15.46:0/632124427 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b04f94bc-5631-478e-904c-1dfc9a673a7e", "id": 3} v 0) -- 0x7f87f0005380 con 0x7f88100a5810 2024-09-16T03:04:39.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.700+0000 7f87f77fe640 1 -- 172.21.15.46:0/632124427 <== mon.1 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b04f94bc-5631-478e-904c-1dfc9a673a7e", "id": 3}]=0 v6) ==== 112+0+2 (unknown 2093045170 0 525046306) 0x7f8808005e00 con 0x7f88100a5810 2024-09-16T03:04:39.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.700+0000 7f87f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b04f94bc-5631-478e-904c-1dfc9a673a7e", "id": 3}] 2024-09-16T03:04:39.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.700+0000 7f87f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:39.706 INFO:teuthology.orchestra.run.smithi046.stdout:3 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 10 librados: watch_flush enter 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 10 librados: watch_flush exit 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 20 client.4126.objecter shutdown clearing up homeless session... 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 10 client.4126.objecter successfully canceled tick 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 10 monclient: shutdown 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 1 -- 172.21.15.46:0/632124427 >> v1:172.21.15.46:6789/0 conn(0x7f88100a5810 legacy=0x7f88101d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 1 -- 172.21.15.46:0/632124427 shutdown_connections 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 1 -- 172.21.15.46:0/632124427 >> 172.21.15.46:0/632124427 conn(0x7f88100aa720 msgr2=0x7f88100a30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 1 -- 172.21.15.46:0/632124427 shutdown_connections 2024-09-16T03:04:39.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 1 -- 172.21.15.46:0/632124427 wait complete. 2024-09-16T03:04:39.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.704+0000 7f8815735640 1 librados: shutdown 2024-09-16T03:04:39.721 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new c2dbca96-afb9-48ed-8a71-8a2bea37d152 4 2024-09-16T03:04:39.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.828+0000 7f26fe910640 10 monclient: get_monmap_and_config 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.828+0000 7f26fe910640 10 monclient: build_initial_monmap 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.828+0000 7f26fe910640 10 monclient: monmap: 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:39.831726+0000 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:39.831726+0000 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:39.833 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.828+0000 7f26fe910640 1 Processor -- start 2024-09-16T03:04:39.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.828+0000 7f26fe910640 1 -- start start 2024-09-16T03:04:39.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.828+0000 7f26fe910640 10 monclient: init 2024-09-16T03:04:39.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:39.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:04:39.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient(hunting): picked mon.noname-a con 0x7f26f80a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:39.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient(hunting): picked mon.noname-c con 0x7f26f80a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:39.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient(hunting): picked mon.noname-b con 0x7f26f8073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:39.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f809d9e0 con 0x7f26f80a4d20 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f809dc20 con 0x7f26f80a5810 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f8073360 con 0x7f26f8073d40 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient(hunting): authenticate will time out at 963.552451s 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26ef7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f26f80a5810 0x7f26f813cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58288/0 (socket says 172.21.15.46:58288) 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26ef7fe640 1 -- 172.21.15.46:0/1138227233 learned_addr learned my addr 172.21.15.46:0/1138227233 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 -- 172.21.15.46:0/1138227233 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3334015619 0 0) 0x7f26f809dc20 con 0x7f26f80a5810 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient(hunting): my global_id is 4129 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 -- 172.21.15.46:0/1138227233 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26e4003530 con 0x7f26f80a5810 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 -- 172.21.15.46:0/1138227233 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4113781442 0 0) 0x7f26e4003530 con 0x7f26f80a5810 2024-09-16T03:04:39.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 -- 172.21.15.46:0/1138227233 >> v1:172.21.15.45:6790/0 conn(0x7f26f8073d40 legacy=0x7f26f8140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 -- 172.21.15.46:0/1138227233 >> v1:172.21.15.45:6789/0 conn(0x7f26f80a4d20 legacy=0x7f26f80a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 monclient: found mon.noname-c 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 -- 172.21.15.46:0/1138227233 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f809cf30 con 0x7f26f80a5810 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: _finish_auth 0 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 5 monclient: authenticate success, global_id 4129 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 -- 172.21.15.46:0/1138227233 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f26e00016a0 con 0x7f26f80a5810 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: dump: 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 -- 172.21.15.46:0/1138227233 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26e00018c0 con 0x7f26f80a5810 2024-09-16T03:04:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 1 -- 172.21.15.46:0/1138227233 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f26e0001d50 con 0x7f26f80a5810 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 4 set_mon_vals no callback set 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26f77fe640 10 monclient: dump: 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient: shutdown 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 -- 172.21.15.46:0/1138227233 >> v1:172.21.15.46:6789/0 conn(0x7f26f80a5810 legacy=0x7f26f813cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 -- 172.21.15.46:0/1138227233 shutdown_connections 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 -- 172.21.15.46:0/1138227233 >> 172.21.15.46:0/1138227233 conn(0x7f26f80aa720 msgr2=0x7f26f80aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 -- 172.21.15.46:0/1138227233 shutdown_connections 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 -- 172.21.15.46:0/1138227233 wait complete. 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient: build_initial_monmap 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient: monmap: 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 librados: starting msgr at 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 librados: starting objecter 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 Processor -- start 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 -- start start 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 librados: setting wanted keys 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 1 librados: calling monclient init 2024-09-16T03:04:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.832+0000 7f26fe910640 10 monclient: init 2024-09-16T03:04:39.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:39.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:04:39.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient(hunting): picked mon.noname-c con 0x7f26f8073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:39.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient(hunting): picked mon.noname-a con 0x7f26f80a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:39.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient(hunting): picked mon.noname-b con 0x7f26f80a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:39.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f8140570 con 0x7f26f80a4d20 2024-09-16T03:04:39.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f813d950 con 0x7f26f80a5810 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f8140d00 con 0x7f26f8073d40 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient(hunting): authenticate will time out at 963.556352s 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fcead640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f26f80a5810 0x7f26f81d5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58298/0 (socket says 172.21.15.46:58298) 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fcead640 1 -- 172.21.15.46:0/2838282459 learned_addr learned my addr 172.21.15.46:0/2838282459 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 client.?.objecter ms_handle_connect 0x7f26f80a5810 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477364029 0 0) 0x7f26f813d950 con 0x7f26f80a5810 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): my global_id is 4132 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d8004ef0 con 0x7f26f80a5810 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 client.?.objecter ms_handle_connect 0x7f26f80a4d20 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 client.?.objecter ms_handle_connect 0x7f26f8073d40 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 718159829 0 0) 0x7f26d8004ef0 con 0x7f26f80a5810 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26f813d950 con 0x7f26f80a5810 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f26f0001d40 con 0x7f26f80a5810 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): dump: 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293256470 0 0) 0x7f26f8140570 con 0x7f26f80a4d20 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): _negotiate 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient(hunting): my global_id is 4125 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d8004ef0 con 0x7f26f80a4d20 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1200160457 0 0) 0x7f26f813d950 con 0x7f26f80a5810 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 >> v1:172.21.15.45:6790/0 conn(0x7f26f8073d40 legacy=0x7f26f813cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 >> v1:172.21.15.45:6789/0 conn(0x7f26f80a4d20 legacy=0x7f26f813d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient: _finish_hunting 0 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 monclient: found mon.b 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f8140850 con 0x7f26f80a5810 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient: _finish_auth 0 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient: _check_auth_tickets 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 5 monclient: authenticate success, global_id 4132 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient: _renew_subs 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 1 -- 172.21.15.46:0/2838282459 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f26f80a1fd0 con 0x7f26f80a5810 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient: _renew_subs 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 1 -- 172.21.15.46:0/2838282459 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f26f80a3cd0 con 0x7f26f80a5810 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 1 librados: init done 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26f0002000 con 0x7f26f80a5810 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f6ffd640 4 set_mon_vals no callback set 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f26f0003df0 con 0x7f26f80a5810 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 monclient: dump: 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f26f00043a0 con 0x7f26f80a5810 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 3629687411 0 0) 0x7f26f0006be0 con 0x7f26f80a5810 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 10 client.4132.objecter ms_dispatch 0x7f26f8000ba0 osd_map(6..6 src has 1..6) 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26f57fa640 20 client.4132.objecter dump_active .. 0 homeless 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.836+0000 7f26fe910640 1 -- 172.21.15.46:0/2838282459 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26dc005090 con 0x7f26f80a5810 2024-09-16T03:04:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.840+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f26f0006610 con 0x7f26f80a5810 2024-09-16T03:04:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.840+0000 7f26f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.840+0000 7f26f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:39.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.912+0000 7f26fe910640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c2dbca96-afb9-48ed-8a71-8a2bea37d152", "id": 4}] 2024-09-16T03:04:39.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.912+0000 7f26fe910640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c2dbca96-afb9-48ed-8a71-8a2bea37d152", "id": 4}] 2024-09-16T03:04:39.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.912+0000 7f26fe910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:39.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.912+0000 7f26fe910640 1 -- 172.21.15.46:0/2838282459 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c2dbca96-afb9-48ed-8a71-8a2bea37d152", "id": 4} v 0) -- 0x7f26dc005380 con 0x7f26f80a5810 2024-09-16T03:04:39.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26f57fa640 1 -- 172.21.15.46:0/2838282459 <== mon.1 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c2dbca96-afb9-48ed-8a71-8a2bea37d152", "id": 4}]=0 v7) ==== 112+0+2 (unknown 4220106326 0 1697012583) 0x7f26f00067f0 con 0x7f26f80a5810 2024-09-16T03:04:39.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c2dbca96-afb9-48ed-8a71-8a2bea37d152", "id": 4}] 2024-09-16T03:04:39.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:39.920 INFO:teuthology.orchestra.run.smithi046.stdout:4 2024-09-16T03:04:39.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 10 librados: watch_flush enter 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 10 librados: watch_flush exit 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 20 client.4132.objecter shutdown clearing up homeless session... 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 10 client.4132.objecter successfully canceled tick 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 10 monclient: shutdown 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 1 -- 172.21.15.46:0/2838282459 >> v1:172.21.15.46:6789/0 conn(0x7f26f80a5810 legacy=0x7f26f81d5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 1 -- 172.21.15.46:0/2838282459 shutdown_connections 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 1 -- 172.21.15.46:0/2838282459 >> 172.21.15.46:0/2838282459 conn(0x7f26f80aa720 msgr2=0x7f26f80a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 1 -- 172.21.15.46:0/2838282459 shutdown_connections 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.916+0000 7f26fe910640 1 -- 172.21.15.46:0/2838282459 wait complete. 2024-09-16T03:04:39.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:39.920+0000 7f26fe910640 1 librados: shutdown 2024-09-16T03:04:39.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new dabb8542-5498-4389-ad9e-0c0a54387723 5 2024-09-16T03:04:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.044+0000 7f45fe9fd640 10 monclient: get_monmap_and_config 2024-09-16T03:04:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.044+0000 7f45fe9fd640 10 monclient: build_initial_monmap 2024-09-16T03:04:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.044+0000 7f45fe9fd640 10 monclient: monmap: 2024-09-16T03:04:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:40.047121+0000 2024-09-16T03:04:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:40.047121+0000 2024-09-16T03:04:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:40.049 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:40.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.044+0000 7f45fe9fd640 1 Processor -- start 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.044+0000 7f45fe9fd640 1 -- start start 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.044+0000 7f45fe9fd640 10 monclient: init 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): picked mon.noname-b con 0x7f45f80a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): picked mon.noname-c con 0x7f45f80a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): picked mon.noname-a con 0x7f45f8073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f809d9e0 con 0x7f45f8073d40 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f809dc20 con 0x7f45f80a5810 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f8073360 con 0x7f45f80a4d20 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): authenticate will time out at 963.768308s 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f45f80a5810 0x7f45f813cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58304/0 (socket says 172.21.15.46:58304) 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f77fe640 1 -- 172.21.15.46:0/2186329794 learned_addr learned my addr 172.21.15.46:0/2186329794 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 -- 172.21.15.46:0/2186329794 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129262880 0 0) 0x7f45f809dc20 con 0x7f45f80a5810 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient(hunting): my global_id is 4135 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 -- 172.21.15.46:0/2186329794 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45e4003530 con 0x7f45f80a5810 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 -- 172.21.15.46:0/2186329794 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3382010762 0 0) 0x7f45e4003530 con 0x7f45f80a5810 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 -- 172.21.15.46:0/2186329794 >> v1:172.21.15.45:6790/0 conn(0x7f45f80a4d20 legacy=0x7f45f80a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 -- 172.21.15.46:0/2186329794 >> v1:172.21.15.45:6789/0 conn(0x7f45f8073d40 legacy=0x7f45f8140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:04:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 monclient: found mon.noname-c 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 -- 172.21.15.46:0/2186329794 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f809cf30 con 0x7f45f80a5810 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 -- 172.21.15.46:0/2186329794 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f45e00016a0 con 0x7f45f80a5810 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: dump: 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 5 monclient: authenticate success, global_id 4135 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 -- 172.21.15.46:0/2186329794 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f45e00018c0 con 0x7f45f80a5810 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 1 -- 172.21.15.46:0/2186329794 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f45e0001d10 con 0x7f45f80a5810 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 4 set_mon_vals no callback set 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f6ffd640 10 monclient: dump: 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient: shutdown 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- 172.21.15.46:0/2186329794 >> v1:172.21.15.46:6789/0 conn(0x7f45f80a5810 legacy=0x7f45f813cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- 172.21.15.46:0/2186329794 shutdown_connections 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- 172.21.15.46:0/2186329794 >> 172.21.15.46:0/2186329794 conn(0x7f45f80aa720 msgr2=0x7f45f80aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- 172.21.15.46:0/2186329794 shutdown_connections 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- 172.21.15.46:0/2186329794 wait complete. 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient: build_initial_monmap 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient: monmap: 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 librados: starting msgr at 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 librados: starting objecter 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 Processor -- start 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- start start 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 librados: setting wanted keys 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 librados: calling monclient init 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient: init 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): picked mon.noname-b con 0x7f45f8073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): picked mon.noname-c con 0x7f45f80a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): picked mon.noname-a con 0x7f45f80a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f8140460 con 0x7f45f80a5810 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f813d790 con 0x7f45f8073d40 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f8138af0 con 0x7f45f80a4d20 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45fe9fd640 10 monclient(hunting): authenticate will time out at 963.771075s 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f45f8073d40 0x7f45f813c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58320/0 (socket says 172.21.15.46:58320) 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f7fff640 1 -- 172.21.15.46:0/184449074 learned_addr learned my addr 172.21.15.46:0/184449074 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 client.?.objecter ms_handle_connect 0x7f45f8073d40 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3036976383 0 0) 0x7f45f813d790 con 0x7f45f8073d40 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): my global_id is 4138 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45d8004ec0 con 0x7f45f8073d40 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 client.?.objecter ms_handle_connect 0x7f45f80a4d20 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 client.?.objecter ms_handle_connect 0x7f45f80a5810 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 485866467 0 0) 0x7f45d8004ec0 con 0x7f45f8073d40 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45f813d790 con 0x7f45f8073d40 2024-09-16T03:04:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f45e80019e0 con 0x7f45f8073d40 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): dump: 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3036363075 0 0) 0x7f45f8140460 con 0x7f45f80a5810 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): my global_id is 4128 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45d8004ec0 con 0x7f45f80a5810 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4191715763 0 0) 0x7f45f8138af0 con 0x7f45f80a4d20 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 10 monclient(hunting): my global_id is 4145 2024-09-16T03:04:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45f8140460 con 0x7f45f80a4d20 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4013192551 0 0) 0x7f45f813d790 con 0x7f45f8073d40 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.048+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 >> v1:172.21.15.45:6790/0 conn(0x7f45f80a4d20 legacy=0x7f45f813d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 >> v1:172.21.15.45:6789/0 conn(0x7f45f80a5810 legacy=0x7f45f81d51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 1 monclient: found mon.b 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f80a1e60 con 0x7f45f8073d40 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: _finish_auth 0 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 5 monclient: authenticate success, global_id 4138 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 10 monclient: _renew_subs 2024-09-16T03:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 1 -- 172.21.15.46:0/184449074 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f45f8140180 con 0x7f45f8073d40 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 10 monclient: _renew_subs 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 1 -- 172.21.15.46:0/184449074 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f45f813ef40 con 0x7f45f8073d40 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 1 librados: init done 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f45e8001d20 con 0x7f45f8073d40 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f45e8003a70 con 0x7f45f8073d40 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f67fc640 4 set_mon_vals no callback set 2024-09-16T03:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: dump: 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f45e8004020 con 0x7f45f8073d40 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 3946872315 0 0) 0x7f45e8005d50 con 0x7f45f8073d40 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 client.4138.objecter ms_dispatch 0x7f45f8000ba0 osd_map(7..7 src has 1..7) 2024-09-16T03:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-09-16T03:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-09-16T03:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-09-16T03:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45fe9fd640 1 -- 172.21.15.46:0/184449074 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45dc005090 con 0x7f45f8073d40 2024-09-16T03:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f45e8004370 con 0x7f45f8073d40 2024-09-16T03:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.052+0000 7f45f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.124+0000 7f45fe9fd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dabb8542-5498-4389-ad9e-0c0a54387723", "id": 5}] 2024-09-16T03:04:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.124+0000 7f45fe9fd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dabb8542-5498-4389-ad9e-0c0a54387723", "id": 5}] 2024-09-16T03:04:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.124+0000 7f45fe9fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:04:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.124+0000 7f45fe9fd640 1 -- 172.21.15.46:0/184449074 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dabb8542-5498-4389-ad9e-0c0a54387723", "id": 5} v 0) -- 0x7f45dc005380 con 0x7f45f8073d40 2024-09-16T03:04:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45f4ff9640 1 -- 172.21.15.46:0/184449074 <== mon.1 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dabb8542-5498-4389-ad9e-0c0a54387723", "id": 5}]=0 v8) ==== 112+0+2 (unknown 2843251927 0 1988412176) 0x7f45e80058a0 con 0x7f45f8073d40 2024-09-16T03:04:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dabb8542-5498-4389-ad9e-0c0a54387723", "id": 5}] 2024-09-16T03:04:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45f4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:40.131 INFO:teuthology.orchestra.run.smithi046.stdout:5 2024-09-16T03:04:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 10 librados: watch_flush enter 2024-09-16T03:04:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 10 librados: watch_flush exit 2024-09-16T03:04:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 20 client.4138.objecter shutdown clearing up homeless session... 2024-09-16T03:04:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 10 client.4138.objecter successfully canceled tick 2024-09-16T03:04:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 10 monclient: shutdown 2024-09-16T03:04:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 1 -- 172.21.15.46:0/184449074 >> v1:172.21.15.46:6789/0 conn(0x7f45f8073d40 legacy=0x7f45f813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 1 -- 172.21.15.46:0/184449074 shutdown_connections 2024-09-16T03:04:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 1 -- 172.21.15.46:0/184449074 >> 172.21.15.46:0/184449074 conn(0x7f45f80aa720 msgr2=0x7f45f81415f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 1 -- 172.21.15.46:0/184449074 shutdown_connections 2024-09-16T03:04:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 1 -- 172.21.15.46:0/184449074 wait complete. 2024-09-16T03:04:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T03:04:40.128+0000 7f45fe9fd640 1 librados: shutdown 2024-09-16T03:04:40.147 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T03:04:40.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T03:04:40.150 INFO:tasks.ceph.osd.0:Started 2024-09-16T03:04:40.150 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T03:04:40.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T03:04:40.153 INFO:tasks.ceph.osd.1:Started 2024-09-16T03:04:40.153 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T03:04:40.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T03:04:40.155 INFO:tasks.ceph.osd.2:Started 2024-09-16T03:04:40.155 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T03:04:40.156 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T03:04:40.157 INFO:tasks.ceph.osd.3:Started 2024-09-16T03:04:40.158 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T03:04:40.158 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T03:04:40.160 INFO:tasks.ceph.osd.4:Started 2024-09-16T03:04:40.160 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T03:04:40.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T03:04:40.162 INFO:tasks.ceph.osd.5:Started 2024-09-16T03:04:40.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.279+0000 7f4109560640 10 monclient: get_monmap_and_config 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.279+0000 7f4109560640 10 monclient: build_initial_monmap 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.279+0000 7f4109560640 10 monclient: monmap: 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:40.283315+0000 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:40.283315+0000 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:40.285 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.279+0000 7f4109560640 1 Processor -- start 2024-09-16T03:04:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- start start 2024-09-16T03:04:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: init 2024-09-16T03:04:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:04:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): picked mon.noname-c con 0x7f4104137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): picked mon.noname-b con 0x7f4104138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): picked mon.noname-a con 0x7f410413e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41040a4f90 con 0x7f410413e2a0 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41040a4cb0 con 0x7f4104137d20 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f410409dc70 con 0x7f4104138100 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): authenticate will time out at 970.932647s 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f41027fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4104138100 0x7f410413cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58950/0 (socket says 172.21.15.45:58950) 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f41037fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f410413e2a0 0x7f4104140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:32876/0 (socket says 172.21.15.45:32876) 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f41027fc640 1 -- 172.21.15.45:0/4292107177 learned_addr learned my addr 172.21.15.45:0/4292107177 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669816260 0 0) 0x7f410409dc70 con 0x7f4104138100 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): my global_id is 4166 2024-09-16T03:04:40.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40ec003530 con 0x7f4104138100 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3370959423 0 0) 0x7f41040a4f90 con 0x7f410413e2a0 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): my global_id is 4149 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f410409dc70 con 0x7f410413e2a0 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3453689913 0 0) 0x7f41040a4cb0 con 0x7f4104137d20 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient(hunting): my global_id is 4159 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41040a4f90 con 0x7f4104137d20 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2679277880 0 0) 0x7f410409dc70 con 0x7f410413e2a0 2024-09-16T03:04:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 >> v1:172.21.15.45:6790/0 conn(0x7f4104138100 legacy=0x7f410413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 >> v1:172.21.15.46:6789/0 conn(0x7f4104137d20 legacy=0x7f41040a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 monclient: found mon.noname-a 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410409cc50 con 0x7f410413e2a0 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: _finish_auth 0 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f40fc001a00 con 0x7f410413e2a0 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 5 monclient: authenticate success, global_id 4149 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: dump: 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40fc001c20 con 0x7f410413e2a0 2024-09-16T03:04:40.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 4 set_mon_vals no callback set 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 1 -- 172.21.15.45:0/4292107177 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f40fc0020b0 con 0x7f410413e2a0 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4101ffb640 10 monclient: dump: 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.297 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: shutdown 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- 172.21.15.45:0/4292107177 >> v1:172.21.15.45:6789/0 conn(0x7f410413e2a0 legacy=0x7f4104140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- 172.21.15.45:0/4292107177 shutdown_connections 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- 172.21.15.45:0/4292107177 >> 172.21.15.45:0/4292107177 conn(0x7f41040aa110 msgr2=0x7f41040aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- 172.21.15.45:0/4292107177 shutdown_connections 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- 172.21.15.45:0/4292107177 wait complete. 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: build_initial_monmap 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: monmap: 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 librados: starting msgr at 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 librados: starting objecter 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 Processor -- start 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- start start 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 librados: setting wanted keys 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 librados: calling monclient init 2024-09-16T03:04:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: init 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): picked mon.noname-a con 0x7f4104137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): picked mon.noname-c con 0x7f4104138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): picked mon.noname-b con 0x7f410413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f410413ad10 con 0x7f4104137d20 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4104141990 con 0x7f410413e2a0 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41041389b0 con 0x7f4104138100 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4109560640 10 monclient(hunting): authenticate will time out at 970.935669s 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4102ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4104137d20 0x7f41041ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:32880/0 (socket says 172.21.15.45:32880) 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f4102ffd640 1 -- 172.21.15.45:0/2278568126 learned_addr learned my addr 172.21.15.45:0/2278568126 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f40eb7fe640 10 client.?.objecter ms_handle_connect 0x7f4104137d20 2024-09-16T03:04:40.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f40eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f40eb7fe640 10 client.?.objecter ms_handle_connect 0x7f4104138100 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f40eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f40eb7fe640 10 client.?.objecter ms_handle_connect 0x7f410413e2a0 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.283+0000 7f40eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276050933 0 0) 0x7f41041389b0 con 0x7f4104138100 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): my global_id is 4169 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40f0004ec0 con 0x7f4104138100 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1275495819 0 0) 0x7f410413ad10 con 0x7f4104137d20 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): my global_id is 4152 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41041389b0 con 0x7f4104137d20 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3204292378 0 0) 0x7f4104141990 con 0x7f410413e2a0 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): my global_id is 4162 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f410413ad10 con 0x7f410413e2a0 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 187004835 0 0) 0x7f40f0004ec0 con 0x7f4104138100 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4104141990 con 0x7f4104138100 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f40f8001850 con 0x7f4104138100 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): dump: 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3602821959 0 0) 0x7f41041389b0 con 0x7f4104137d20 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40f0004ec0 con 0x7f4104137d20 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f40f4001a60 con 0x7f4104137d20 2024-09-16T03:04:40.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): dump: 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.304 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1012444583 0 0) 0x7f410413ad10 con 0x7f410413e2a0 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41041389b0 con 0x7f410413e2a0 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f40fc001dd0 con 0x7f410413e2a0 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient(hunting): dump: 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2343882870 0 0) 0x7f4104141990 con 0x7f4104138100 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 >> v1:172.21.15.46:6789/0 conn(0x7f410413e2a0 legacy=0x7f41041d5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 >> v1:172.21.15.45:6789/0 conn(0x7f4104137d20 legacy=0x7f41041ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 monclient: found mon.c 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:40.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410413aa30 con 0x7f4104138100 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 5 monclient: authenticate success, global_id 4169 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 10 monclient: _renew_subs 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4104139510 con 0x7f4104138100 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 10 monclient: _renew_subs 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f41040a3df0 con 0x7f4104138100 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 1 librados: init done 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40f80047f0 con 0x7f4104138100 2024-09-16T03:04:40.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f40f8004b40 con 0x7f4104138100 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f41017fa640 4 set_mon_vals no callback set 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: dump: 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f40f8004ed0 con 0x7f4104138100 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 1417608888 0 0) 0x7f40f8006650 con 0x7f4104138100 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 client.4169.objecter ms_dispatch 0x7f4104000ba0 osd_map(8..8 src has 1..8) 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 3 client.4169.objecter handle_osd_map got epochs [8,8] > 0 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 3 client.4169.objecter handle_osd_map decoding full epoch 8 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 20 client.4169.objecter dump_active .. 0 homeless 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f4109560640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e4005090 con 0x7f4104138100 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f40f8006080 con 0x7f4104138100 2024-09-16T03:04:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:40.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.287+0000 7f40eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:40.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:40.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:40.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:40.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 1 -- 172.21.15.45:0/2278568126 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40e4005380 con 0x7f4104138100 2024-09-16T03:04:40.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f40eb7fe640 1 -- 172.21.15.45:0/2278568126 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 4129073490) 0x7f40f80063d0 con 0x7f4104138100 2024-09-16T03:04:40.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f40eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:40.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f40eb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:40.355 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:40.355 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","created":"2024-09-16T03:04:38.421437+0000","modified":"2024-09-16T03:04:40.127320+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:04:40.127320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"60a0a607-fe6c-4bd5-b369-e0dd6f6bd551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"10adca9c-3542-4901-a439-a1c9e9fcb8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88fe1895-098a-4f13-95b2-4ead1739e71c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b04f94bc-5631-478e-904c-1dfc9a673a7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c2dbca96-afb9-48ed-8a71-8a2bea37d152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dabb8542-5498-4389-ad9e-0c0a54387723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:04:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 10 librados: watch_flush enter 2024-09-16T03:04:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 10 librados: watch_flush exit 2024-09-16T03:04:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 20 client.4169.objecter shutdown clearing up homeless session... 2024-09-16T03:04:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 10 client.4169.objecter successfully canceled tick 2024-09-16T03:04:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 10 monclient: shutdown 2024-09-16T03:04:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:40.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 1 -- 172.21.15.45:0/2278568126 >> v1:172.21.15.45:6790/0 conn(0x7f4104138100 legacy=0x7f41041d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:40.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 1 -- 172.21.15.45:0/2278568126 shutdown_connections 2024-09-16T03:04:40.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 1 -- 172.21.15.45:0/2278568126 >> 172.21.15.45:0/2278568126 conn(0x7f41040aa110 msgr2=0x7f4104130960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:40.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 1 -- 172.21.15.45:0/2278568126 shutdown_connections 2024-09-16T03:04:40.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 1 -- 172.21.15.45:0/2278568126 wait complete. 2024-09-16T03:04:40.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:40.351+0000 7f4109560640 1 librados: shutdown 2024-09-16T03:04:40.372 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T03:04:40.372 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T03:04:40.977 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:04:41.100 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.095+0000 7f525bc49640 10 monclient: get_monmap_and_config 2024-09-16T03:04:41.100 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.095+0000 7f525bc49640 10 monclient: build_initial_monmap 2024-09-16T03:04:41.100 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.095+0000 7f525bc49640 10 monclient: monmap: 2024-09-16T03:04:41.100 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-09-16T03:04:41.100 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:41.100 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:41.099603+0000 2024-09-16T03:04:41.101 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:41.099603+0000 2024-09-16T03:04:41.101 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:41.101 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:41.101 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:41.101 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:41.101 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:41.101 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:41.104 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 Processor -- start 2024-09-16T03:04:41.104 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- start start 2024-09-16T03:04:41.104 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient: init 2024-09-16T03:04:41.104 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:41.104 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient(hunting): picked mon.noname-a con 0x7f525409ed80 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient(hunting): picked mon.noname-b con 0x7f52540a0cc0 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient(hunting): picked mon.noname-c con 0x7f525413bb70 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52540a5b30 con 0x7f525409ed80 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52540a6390 con 0x7f525413bb70 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52540a0100 con 0x7f52540a0cc0 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient(hunting): authenticate will time out at 971.749174s 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52591e4640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f52540a0cc0 0x7f52540a5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:51634/0 (socket says 172.21.15.45:51634) 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52591e4640 1 -- 172.21.15.45:0/269729003 learned_addr learned my addr 172.21.15.45:0/269729003 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1044051157 0 0) 0x7f52540a0100 con 0x7f52540a0cc0 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient(hunting): _negotiate 2024-09-16T03:04:41.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient(hunting): my global_id is 4172 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5240003530 con 0x7f52540a0cc0 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764399972 0 0) 0x7f52540a5b30 con 0x7f525409ed80 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient(hunting): _negotiate 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient(hunting): my global_id is 4155 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52540a0100 con 0x7f525409ed80 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1184380541 0 0) 0x7f52540a0100 con 0x7f525409ed80 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 >> v1:172.21.15.45:6790/0 conn(0x7f52540a0cc0 legacy=0x7f52540a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 >> v1:172.21.15.46:6789/0 conn(0x7f525413bb70 legacy=0x7f525413df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:41.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: _finish_hunting 0 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 monclient: found mon.noname-a 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525409fe20 con 0x7f525409ed80 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: _finish_auth 0 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: _check_auth_tickets 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 5 monclient: authenticate success, global_id 4155 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5250001680 con 0x7f525409ed80 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: dump: 2024-09-16T03:04:41.107 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5250001820 con 0x7f525409ed80 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:41.108 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 1 -- 172.21.15.45:0/269729003 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5250001c30 con 0x7f525409ed80 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 4 set_mon_vals no callback set 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f52589e3640 10 monclient: dump: 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:41.109 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient: shutdown 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- 172.21.15.45:0/269729003 >> v1:172.21.15.45:6789/0 conn(0x7f525409ed80 legacy=0x7f52540a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- 172.21.15.45:0/269729003 shutdown_connections 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- 172.21.15.45:0/269729003 >> 172.21.15.45:0/269729003 conn(0x7f52540a3a80 msgr2=0x7f52540a3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- 172.21.15.45:0/269729003 shutdown_connections 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- 172.21.15.45:0/269729003 wait complete. 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient: build_initial_monmap 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient: monmap: 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:41.110 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 librados: starting msgr at 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 librados: starting objecter 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 Processor -- start 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 -- start start 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 librados: setting wanted keys 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 1 librados: calling monclient init 2024-09-16T03:04:41.111 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.099+0000 7f525bc49640 10 monclient: init 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient(hunting): picked mon.noname-c con 0x7f525409ed80 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient(hunting): picked mon.noname-b con 0x7f52540a0cc0 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient(hunting): picked mon.noname-a con 0x7f525413bb70 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5254137a30 con 0x7f525413bb70 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52541cda20 con 0x7f52540a0cc0 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52540a31e0 con 0x7f525409ed80 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient(hunting): authenticate will time out at 971.752302s 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f52599e5640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f525409ed80 0x7f52541ccc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:51648/0 (socket says 172.21.15.45:51648) 2024-09-16T03:04:41.115 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f52599e5640 1 -- 172.21.15.45:0/3567199830 learned_addr learned my addr 172.21.15.45:0/3567199830 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 client.?.objecter ms_handle_connect 0x7f525413bb70 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 client.?.objecter ms_handle_connect 0x7f525409ed80 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 394916123 0 0) 0x7f5254137a30 con 0x7f525413bb70 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): my global_id is 4158 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5238004ef0 con 0x7f525413bb70 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108707446 0 0) 0x7f52540a31e0 con 0x7f525409ed80 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:41.116 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): my global_id is 4175 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5254137a30 con 0x7f525409ed80 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1128489817 0 0) 0x7f5238004ef0 con 0x7f525413bb70 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52540a31e0 con 0x7f525413bb70 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f524c001c40 con 0x7f525413bb70 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): dump: 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:41.117 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 client.?.objecter ms_handle_connect 0x7f52540a0cc0 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3615566775 0 0) 0x7f5254137a30 con 0x7f525409ed80 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5238004ef0 con 0x7f525409ed80 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5250001c70 con 0x7f525409ed80 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:41.118 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient(hunting): dump: 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2415685772 0 0) 0x7f52540a31e0 con 0x7f525413bb70 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 >> v1:172.21.15.45:6790/0 conn(0x7f525409ed80 legacy=0x7f52541ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 >> v1:172.21.15.46:6789/0 conn(0x7f52540a0cc0 legacy=0x7f52541cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:04:41.119 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 monclient: found mon.a 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52540a2e40 con 0x7f525413bb70 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 5 monclient: authenticate success, global_id 4158 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient: _renew_subs 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 1 -- 172.21.15.45:0/3567199830 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52541361e0 con 0x7f525413bb70 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient: _renew_subs 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:41.120 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 1 -- 172.21.15.45:0/3567199830 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5254134fc0 con 0x7f525413bb70 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 1 librados: init done 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f524c003980 con 0x7f525413bb70 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f524c003cd0 con 0x7f525413bb70 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524bfff640 4 set_mon_vals no callback set 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 monclient: dump: 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:41.121 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f524c004200 con 0x7f525413bb70 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 1417608888 0 0) 0x7f524c005a70 con 0x7f525413bb70 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 10 client.4158.objecter ms_dispatch 0x7f5254000ba0 osd_map(8..8 src has 1..8) 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 3 client.4158.objecter handle_osd_map got epochs [8,8] > 0 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 3 client.4158.objecter handle_osd_map decoding full epoch 8 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f524a7fc640 20 client.4158.objecter dump_active .. 0 homeless 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:41.122 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.103+0000 7f525bc49640 1 -- 172.21.15.45:0/3567199830 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f523c005090 con 0x7f525413bb70 2024-09-16T03:04:41.123 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.107+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f524c005530 con 0x7f525413bb70 2024-09-16T03:04:41.123 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.107+0000 7f524a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:41.123 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.107+0000 7f524a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:41.170 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:41.170 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:41.170 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:41.170 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 1 -- 172.21.15.45:0/3567199830 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f523c005380 con 0x7f525413bb70 2024-09-16T03:04:41.171 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f524a7fc640 1 -- 172.21.15.45:0/3567199830 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 4129073490) 0x7f524c005840 con 0x7f525413bb70 2024-09-16T03:04:41.171 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f524a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:41.171 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f524a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:41.171 INFO:teuthology.misc.health.smithi045.stdout: 2024-09-16T03:04:41.171 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":8,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","created":"2024-09-16T03:04:38.421437+0000","modified":"2024-09-16T03:04:40.127320+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:04:40.127320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"60a0a607-fe6c-4bd5-b369-e0dd6f6bd551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"10adca9c-3542-4901-a439-a1c9e9fcb8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88fe1895-098a-4f13-95b2-4ead1739e71c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b04f94bc-5631-478e-904c-1dfc9a673a7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c2dbca96-afb9-48ed-8a71-8a2bea37d152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dabb8542-5498-4389-ad9e-0c0a54387723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:04:41.172 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 10 librados: watch_flush enter 2024-09-16T03:04:41.172 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 10 librados: watch_flush exit 2024-09-16T03:04:41.172 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 20 client.4158.objecter shutdown clearing up homeless session... 2024-09-16T03:04:41.172 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 10 client.4158.objecter successfully canceled tick 2024-09-16T03:04:41.172 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 10 monclient: shutdown 2024-09-16T03:04:41.173 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:41.173 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 1 -- 172.21.15.45:0/3567199830 >> v1:172.21.15.45:6789/0 conn(0x7f525413bb70 legacy=0x7f52541d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:41.173 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 1 -- 172.21.15.45:0/3567199830 shutdown_connections 2024-09-16T03:04:41.173 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 1 -- 172.21.15.45:0/3567199830 >> 172.21.15.45:0/3567199830 conn(0x7f52540a3a80 msgr2=0x7f5254073310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:41.173 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 1 -- 172.21.15.45:0/3567199830 shutdown_connections 2024-09-16T03:04:41.173 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 1 -- 172.21.15.45:0/3567199830 wait complete. 2024-09-16T03:04:41.173 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:41.167+0000 7f525bc49640 1 librados: shutdown 2024-09-16T03:04:41.187 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T03:04:41.314 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-09-16T03:04:41.311+0000 7f09ac34a7c0 -1 Falling back to public interface 2024-09-16T03:04:41.418 INFO:tasks.ceph.mgr.x.smithi046.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T03:04:41.418 INFO:tasks.ceph.mgr.x.smithi046.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T03:04:41.418 INFO:tasks.ceph.mgr.x.smithi046.stderr: from numpy import show_config as show_numpy_config 2024-09-16T03:04:41.518 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-09-16T03:04:41.515+0000 7fb6defb17c0 -1 Falling back to public interface 2024-09-16T03:04:41.531 INFO:tasks.ceph.osd.4.smithi046.stderr:2024-09-16T03:04:41.528+0000 7f9693abf7c0 -1 Falling back to public interface 2024-09-16T03:04:41.578 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-09-16T03:04:41.575+0000 7f637052b7c0 -1 Falling back to public interface 2024-09-16T03:04:41.735 INFO:tasks.ceph.osd.5.smithi046.stderr:2024-09-16T03:04:41.732+0000 7f0b7bc9f7c0 -1 Falling back to public interface 2024-09-16T03:04:41.796 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-09-16T03:04:41.792+0000 7f93448727c0 -1 Falling back to public interface 2024-09-16T03:04:44.464 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-09-16T03:04:44.460+0000 7f8984570640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T03:04:44.466 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-09-16T03:04:44.464+0000 7f8984570640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T03:04:45.270 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-09-16T03:04:45.267+0000 7f09ac34a7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T03:04:45.455 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-09-16T03:04:45.451+0000 7fb6defb17c0 -1 osd.2 0 log_to_monitors true 2024-09-16T03:04:45.618 INFO:tasks.ceph.osd.4.smithi046.stderr:2024-09-16T03:04:45.616+0000 7f9693abf7c0 -1 osd.4 0 log_to_monitors true 2024-09-16T03:04:45.810 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-09-16T03:04:45.807+0000 7f637052b7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T03:04:45.820 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-09-16T03:04:45.816+0000 7f93448727c0 -1 osd.3 0 log_to_monitors true 2024-09-16T03:04:46.015 INFO:tasks.ceph.osd.5.smithi046.stderr:2024-09-16T03:04:46.012+0000 7f0b7bc9f7c0 -1 osd.5 0 log_to_monitors true 2024-09-16T03:04:47.427 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-09-16T03:04:47.423+0000 7fb6daf5a640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T03:04:47.431 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-09-16T03:04:47.427+0000 7fb6d2d8c640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:04:47.444 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-09-16T03:04:47.439+0000 7f09a82f3640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T03:04:47.447 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-09-16T03:04:47.443+0000 7f09a0125640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:04:47.539 INFO:tasks.ceph.osd.4.smithi046.stderr:2024-09-16T03:04:47.536+0000 7f968fa68640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T03:04:47.548 INFO:tasks.ceph.osd.4.smithi046.stderr:2024-09-16T03:04:47.544+0000 7f968b8a2640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:04:47.791 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:04:47.910 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient: get_monmap_and_config 2024-09-16T03:04:47.910 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient: build_initial_monmap 2024-09-16T03:04:47.911 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient: monmap: 2024-09-16T03:04:47.911 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-09-16T03:04:47.911 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:47.911 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:47.909003+0000 2024-09-16T03:04:47.911 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:47.909003+0000 2024-09-16T03:04:47.912 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:47.912 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:47.912 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:47.912 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:47.912 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:47.912 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:47.913 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 1 Processor -- start 2024-09-16T03:04:47.913 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 1 -- start start 2024-09-16T03:04:47.913 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient: init 2024-09-16T03:04:47.925 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:47.925 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:04:47.925 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient(hunting): picked mon.noname-c con 0x7f91cc0a4fb0 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:47.925 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient(hunting): picked mon.noname-b con 0x7f91cc06d780 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:47.925 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient(hunting): picked mon.noname-a con 0x7f91cc0a5aa0 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:47.926 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91cc06d280 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.926 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91cc06ccc0 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.926 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91cc0a6db0 con 0x7f91cc06d780 2024-09-16T03:04:47.926 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:47.926 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient(hunting): authenticate will time out at 978.558611s 2024-09-16T03:04:47.927 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d0b80640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f91cc06d780 0x7f91cc13d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:51692/0 (socket says 172.21.15.45:51692) 2024-09-16T03:04:47.927 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d0b80640 1 -- 172.21.15.45:0/1871323279 learned_addr learned my addr 172.21.15.45:0/1871323279 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:47.927 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1561624787 0 0) 0x7f91cc0a6db0 con 0x7f91cc06d780 2024-09-16T03:04:47.927 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:04:47.927 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:47.928 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:47.928 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): my global_id is 4214 2024-09-16T03:04:47.928 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91b8003530 con 0x7f91cc06d780 2024-09-16T03:04:47.928 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1415774877 0 0) 0x7f91cc06ccc0 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.928 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:04:47.929 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:47.929 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:47.929 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): my global_id is 4207 2024-09-16T03:04:47.929 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91cc0a6db0 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.929 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3796540855 0 0) 0x7f91cc06d280 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.930 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:04:47.930 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:47.930 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:47.930 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient(hunting): my global_id is 4191 2024-09-16T03:04:47.930 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91cc06ccc0 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.930 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3261516672 0 0) 0x7f91cc06ccc0 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.931 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 >> v1:172.21.15.45:6790/0 conn(0x7f91cc06d780 legacy=0x7f91cc13d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:47.931 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 >> v1:172.21.15.46:6789/0 conn(0x7f91cc0a4fb0 legacy=0x7f91cc0a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:47.931 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:04:47.931 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 monclient: found mon.noname-a 2024-09-16T03:04:47.931 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:47.932 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:04:47.932 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91cc09dcc0 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.932 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: _finish_auth 0 2024-09-16T03:04:47.932 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:04:47.932 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:47.933 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f91c4001940 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.933 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:47.933 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:47.933 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: dump: 2024-09-16T03:04:47.933 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:47.933 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:47.934 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.934 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.934 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:47.934 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:47.934 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:47.935 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:47.935 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:47.935 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:47.935 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 5 monclient: authenticate success, global_id 4191 2024-09-16T03:04:47.935 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:47.935 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f91c4001b60 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.936 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:47.936 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 1 -- 172.21.15.45:0/1871323279 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f91c4001ff0 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.936 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:47.936 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:47.936 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91c3fff640 10 monclient: dump: 2024-09-16T03:04:47.937 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:47.937 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:47.937 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.937 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.937 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:47.937 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:47.938 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:47.938 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:47.938 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:47.938 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:47.938 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:47.939 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.907+0000 7f91d35e5640 4 set_mon_vals no callback set 2024-09-16T03:04:47.939 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient: shutdown 2024-09-16T03:04:47.939 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:47.939 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 -- 172.21.15.45:0/1871323279 >> v1:172.21.15.45:6789/0 conn(0x7f91cc0a5aa0 legacy=0x7f91cc1408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:47.939 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 -- 172.21.15.45:0/1871323279 shutdown_connections 2024-09-16T03:04:47.939 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 -- 172.21.15.45:0/1871323279 >> 172.21.15.45:0/1871323279 conn(0x7f91cc0a7280 msgr2=0x7f91cc0a7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:47.940 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 -- 172.21.15.45:0/1871323279 shutdown_connections 2024-09-16T03:04:47.940 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 -- 172.21.15.45:0/1871323279 wait complete. 2024-09-16T03:04:47.940 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient: build_initial_monmap 2024-09-16T03:04:47.940 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient: monmap: 2024-09-16T03:04:47.940 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-09-16T03:04:47.941 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:47.941 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:47.941 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-09-16T03:04:47.941 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:47.942 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:47.942 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:47.942 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:47.942 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:47.942 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:47.943 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 librados: starting msgr at 2024-09-16T03:04:47.943 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 librados: starting objecter 2024-09-16T03:04:47.943 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 Processor -- start 2024-09-16T03:04:47.943 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 -- start start 2024-09-16T03:04:47.943 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 librados: setting wanted keys 2024-09-16T03:04:47.943 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 librados: calling monclient init 2024-09-16T03:04:47.944 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient: init 2024-09-16T03:04:47.944 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:47.944 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:04:47.944 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient(hunting): picked mon.noname-a con 0x7f91cc06d780 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:47.944 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient(hunting): picked mon.noname-c con 0x7f91cc0a4fb0 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:47.945 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient(hunting): picked mon.noname-b con 0x7f91cc0a5aa0 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:47.945 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91cc140540 con 0x7f91cc06d780 2024-09-16T03:04:47.945 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91cc140820 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.945 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91cc0a1f60 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.945 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:47.945 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d35e5640 10 monclient(hunting): authenticate will time out at 978.562593s 2024-09-16T03:04:47.946 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d1381640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f91cc06d780 0x7f91cc13ccf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41210/0 (socket says 172.21.15.45:41210) 2024-09-16T03:04:47.946 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91d1381640 1 -- 172.21.15.45:0/4017134030 learned_addr learned my addr 172.21.15.45:0/4017134030 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:47.946 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 client.?.objecter ms_handle_connect 0x7f91cc0a4fb0 2024-09-16T03:04:47.946 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:47.946 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 client.?.objecter ms_handle_connect 0x7f91cc06d780 2024-09-16T03:04:47.946 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:47.947 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 client.?.objecter ms_handle_connect 0x7f91cc0a5aa0 2024-09-16T03:04:47.947 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:47.947 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1770908802 0 0) 0x7f91cc0a1f60 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.947 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:47.947 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:47.947 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:47.948 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): my global_id is 4217 2024-09-16T03:04:47.948 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91b0004ec0 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.948 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3494640080 0 0) 0x7f91cc140820 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.948 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:47.948 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:47.948 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:47.949 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): my global_id is 4210 2024-09-16T03:04:47.949 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91cc0a1f60 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.949 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729588798 0 0) 0x7f91cc140540 con 0x7f91cc06d780 2024-09-16T03:04:47.949 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:47.949 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:47.950 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:47.950 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): my global_id is 4194 2024-09-16T03:04:47.950 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91cc140820 con 0x7f91cc06d780 2024-09-16T03:04:47.950 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 111766242 0 0) 0x7f91b0004ec0 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.950 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91cc140540 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.950 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f91c8001960 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.951 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:47.951 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:47.951 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): dump: 2024-09-16T03:04:47.951 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:47.951 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:47.951 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.952 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.952 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:47.952 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:47.952 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:47.952 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:47.953 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:47.953 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:47.953 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:47.953 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3565701903 0 0) 0x7f91cc140820 con 0x7f91cc06d780 2024-09-16T03:04:47.953 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91b0004ec0 con 0x7f91cc06d780 2024-09-16T03:04:47.953 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 52714835 0 0) 0x7f91cc0a1f60 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.954 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91cc140820 con 0x7f91cc0a5aa0 2024-09-16T03:04:47.954 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f91bc001940 con 0x7f91cc06d780 2024-09-16T03:04:47.954 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:47.954 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:47.954 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient(hunting): dump: 2024-09-16T03:04:47.954 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:47.955 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:47.955 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.955 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.955 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:47.955 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:47.955 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:47.956 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:47.956 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:47.956 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:47.956 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2058830264 0 0) 0x7f91cc140540 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.956 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 >> v1:172.21.15.46:6789/0 conn(0x7f91cc0a5aa0 legacy=0x7f91cc1d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:47.956 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 >> v1:172.21.15.45:6789/0 conn(0x7f91cc06d780 legacy=0x7f91cc13ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:47.957 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:04:47.957 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 monclient: found mon.c 2024-09-16T03:04:47.957 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:47.957 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:47.957 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91cc0a17a0 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.958 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:04:47.958 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:04:47.958 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.911+0000 7f91c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:47.958 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 5 monclient: authenticate success, global_id 4217 2024-09-16T03:04:47.958 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 10 monclient: _renew_subs 2024-09-16T03:04:47.958 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:47.959 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f91cc140260 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.959 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:47.959 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 10 monclient: _renew_subs 2024-09-16T03:04:47.959 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:47.959 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f91cc13db10 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.960 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 1 librados: init done 2024-09-16T03:04:47.960 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f91c80047f0 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.960 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:47.960 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f91c8004b40 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.960 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:47.961 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:47.961 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c37fe640 4 set_mon_vals no callback set 2024-09-16T03:04:47.961 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 10 monclient: dump: 2024-09-16T03:04:47.961 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:47.961 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:47.962 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.962 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:47.962 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:47.962 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:47.962 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:47.963 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:47.963 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:47.963 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:47.963 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) ==== 48776+0+0 (unknown 877191757 0 0) 0x7f91c80109d0 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.963 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:47.963 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:47.964 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:47.964 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ac005090 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.964 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) ==== 2207+0+0 (unknown 2317306357 0 0) 0x7f91c804a760 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.964 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 10 client.4217.objecter ms_dispatch 0x7f91cc000ba0 osd_map(11..11 src has 1..11) 2024-09-16T03:04:47.965 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 3 client.4217.objecter handle_osd_map got epochs [11,11] > 0 2024-09-16T03:04:47.965 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 3 client.4217.objecter handle_osd_map decoding full epoch 11 2024-09-16T03:04:47.965 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 20 client.4217.objecter dump_active .. 0 homeless 2024-09-16T03:04:47.965 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91d1381640 10 monclient: get_auth_request con 0x7f91b003e2c0 auth_method 0 2024-09-16T03:04:47.965 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.915+0000 7f91c1ffb640 10 client.4217.objecter ms_handle_connect 0x7f91b003e2c0 2024-09-16T03:04:47.966 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.923+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f91c80166d0 con 0x7f91cc0a4fb0 2024-09-16T03:04:47.966 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.923+0000 7f91c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:47.966 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:47.923+0000 7f91c1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:48.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.103+0000 7f91d35e5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:48.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.103+0000 7f91d35e5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:48.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.103+0000 7f91d35e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:48.105 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.103+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91ac005380 con 0x7f91cc0a4fb0 2024-09-16T03:04:48.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.103+0000 7f91c1ffb640 1 -- 172.21.15.45:0/4017134030 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+6089 (unknown 238102980 0 2627698162) 0x7f91c801aab0 con 0x7f91cc0a4fb0 2024-09-16T03:04:48.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.103+0000 7f91c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:48.106 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.103+0000 7f91c1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:48.106 INFO:teuthology.misc.health.smithi045.stdout: 2024-09-16T03:04:48.106 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","created":"2024-09-16T03:04:38.421437+0000","modified":"2024-09-16T03:04:47.517117+0000","last_up_change":"2024-09-16T03:04:47.517117+0000","last_in_change":"2024-09-16T03:04:40.127320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"60a0a607-fe6c-4bd5-b369-e0dd6f6bd551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6800","nonce":3131549395}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6801","nonce":3131549395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6803","nonce":3131549395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6802","nonce":3131549395}]},"public_addr":"172.21.15.45:6800/3131549395","cluster_addr":"172.21.15.45:6801/3131549395","heartbeat_back_addr":"172.21.15.45:6803/3131549395","heartbeat_front_addr":"172.21.15.45:6802/3131549395","state":["exists","up"]},{"osd":1,"uuid":"10adca9c-3542-4901-a439-a1c9e9fcb8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88fe1895-098a-4f13-95b2-4ead1739e71c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6804","nonce":207876738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6805","nonce":207876738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6807","nonce":207876738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6806","nonce":207876738}]},"public_addr":"172.21.15.45:6804/207876738","cluster_addr":"172.21.15.45:6805/207876738","heartbeat_back_addr":"172.21.15.45:6807/207876738","heartbeat_front_addr":"172.21.15.45:6806/207876738","state":["exists","up"]},{"osd":3,"uuid":"b04f94bc-5631-478e-904c-1dfc9a673a7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c2dbca96-afb9-48ed-8a71-8a2bea37d152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dabb8542-5498-4389-ad9e-0c0a54387723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:04:48.109 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 10 librados: watch_flush enter 2024-09-16T03:04:48.109 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 10 librados: watch_flush exit 2024-09-16T03:04:48.109 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 20 client.4217.objecter shutdown clearing up homeless session... 2024-09-16T03:04:48.109 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 10 client.4217.objecter successfully canceled tick 2024-09-16T03:04:48.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 >> v1:172.21.15.46:6812/149360280 conn(0x7f91b003e2c0 legacy=0x7f91b0040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:48.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 10 monclient: shutdown 2024-09-16T03:04:48.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:48.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 >> v1:172.21.15.45:6790/0 conn(0x7f91cc0a4fb0 legacy=0x7f91cc13d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:48.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 shutdown_connections 2024-09-16T03:04:48.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 >> 172.21.15.45:0/4017134030 conn(0x7f91cc0a7280 msgr2=0x7f91cc0a32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:48.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 shutdown_connections 2024-09-16T03:04:48.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 1 -- 172.21.15.45:0/4017134030 wait complete. 2024-09-16T03:04:48.110 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:48.107+0000 7f91d35e5640 1 librados: shutdown 2024-09-16T03:04:48.127 DEBUG:teuthology.misc:2 of 6 OSDs are up 2024-09-16T03:04:48.246 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-09-16T03:04:48.243+0000 7f636cce6640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T03:04:48.251 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-09-16T03:04:48.247+0000 7f6364306640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:04:48.425 INFO:tasks.ceph.osd.5.smithi046.stderr:2024-09-16T03:04:48.420+0000 7f0b77c48640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T03:04:48.428 INFO:tasks.ceph.osd.5.smithi046.stderr:2024-09-16T03:04:48.424+0000 7f0b6fa7a640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:04:48.433 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-09-16T03:04:48.432+0000 7f934081b640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T03:04:48.436 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-09-16T03:04:48.432+0000 7f933864d640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:04:54.733 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:04:54.856 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.851+0000 7f7898dac640 10 monclient: get_monmap_and_config 2024-09-16T03:04:54.856 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.851+0000 7f7898dac640 10 monclient: build_initial_monmap 2024-09-16T03:04:54.856 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.851+0000 7f7898dac640 10 monclient: monmap: 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:54.855817+0000 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:54.855817+0000 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:54.857 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:54.864 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 Processor -- start 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- start start 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient: init 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient(hunting): picked mon.noname-c con 0x7f78940a1960 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient(hunting): picked mon.noname-a con 0x7f789409dd50 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient(hunting): picked mon.noname-b con 0x7f78940a2450 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789409d650 con 0x7f789409dd50 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789409edf0 con 0x7f78940a1960 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789409f0d0 con 0x7f78940a2450 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient(hunting): authenticate will time out at 985.505149s 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7891d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f789409dd50 0x7f789413ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38698/0 (socket says 172.21.15.45:38698) 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7891d9b640 1 -- 172.21.15.45:0/649571239 learned_addr learned my addr 172.21.15.45:0/649571239 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 935912307 0 0) 0x7f789409d650 con 0x7f789409dd50 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient(hunting): _negotiate 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient(hunting): my global_id is 4206 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f787c003530 con 0x7f789409dd50 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1923612175 0 0) 0x7f789409f0d0 con 0x7f78940a2450 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient(hunting): _negotiate 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient(hunting): my global_id is 4229 2024-09-16T03:04:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f789409d650 con 0x7f78940a2450 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2755363330 0 0) 0x7f789409d650 con 0x7f78940a2450 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 >> v1:172.21.15.46:6789/0 conn(0x7f78940a1960 legacy=0x7f78940a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 >> v1:172.21.15.45:6789/0 conn(0x7f789409dd50 legacy=0x7f789413ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: _finish_hunting 0 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 monclient: found mon.noname-b 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789409f3b0 con 0x7f78940a2450 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: _finish_auth 0 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: _check_auth_tickets 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f788c001a00 con 0x7f78940a2450 2024-09-16T03:04:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 5 monclient: authenticate success, global_id 4229 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: dump: 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:54.868 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f788c001ba0 con 0x7f78940a2450 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 1 -- 172.21.15.45:0/649571239 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f788c001fb0 con 0x7f78940a2450 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 4 set_mon_vals no callback set 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f789159a640 10 monclient: dump: 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.869 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient: shutdown 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- 172.21.15.45:0/649571239 >> v1:172.21.15.45:6790/0 conn(0x7f78940a2450 legacy=0x7f789413e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- 172.21.15.45:0/649571239 shutdown_connections 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- 172.21.15.45:0/649571239 >> 172.21.15.45:0/649571239 conn(0x7f78940a6580 msgr2=0x7f78940a2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- 172.21.15.45:0/649571239 shutdown_connections 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- 172.21.15.45:0/649571239 wait complete. 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient: build_initial_monmap 2024-09-16T03:04:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient: monmap: 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 librados: starting msgr at 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 librados: starting objecter 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 Processor -- start 2024-09-16T03:04:54.871 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 -- start start 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 librados: setting wanted keys 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 1 librados: calling monclient init 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.855+0000 7f7898dac640 10 monclient: init 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient(hunting): picked mon.noname-c con 0x7f789409dd50 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient(hunting): picked mon.noname-a con 0x7f78940a1960 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient(hunting): picked mon.noname-b con 0x7f78940a2450 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789411fd00 con 0x7f78940a1960 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789411de40 con 0x7f78940a2450 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789413b1b0 con 0x7f789409dd50 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient(hunting): authenticate will time out at 985.508491s 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f789259c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f789409dd50 0x7f78941ccda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36084/0 (socket says 172.21.15.45:36084) 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f789259c640 1 -- 172.21.15.45:0/3325666228 learned_addr learned my addr 172.21.15.45:0/3325666228 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 client.?.objecter ms_handle_connect 0x7f78940a1960 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 client.?.objecter ms_handle_connect 0x7f789409dd50 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3863015798 0 0) 0x7f789411fd00 con 0x7f78940a1960 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): my global_id is 4209 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7874004ec0 con 0x7f78940a1960 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369989342 0 0) 0x7f789413b1b0 con 0x7f789409dd50 2024-09-16T03:04:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): my global_id is 4232 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f789411fd00 con 0x7f789409dd50 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 client.?.objecter ms_handle_connect 0x7f78940a2450 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 428901261 0 0) 0x7f7874004ec0 con 0x7f78940a1960 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f789413b1b0 con 0x7f78940a1960 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f78880012e0 con 0x7f78940a1960 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): dump: 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:54.874 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1456020879 0 0) 0x7f789411fd00 con 0x7f789409dd50 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7874004ec0 con 0x7f789409dd50 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7884001920 con 0x7f789409dd50 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:54.875 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): dump: 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3305685515 0 0) 0x7f789411de40 con 0x7f78940a2450 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:54.876 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient(hunting): my global_id is 4225 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f789411fd00 con 0x7f78940a2450 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 292132665 0 0) 0x7f789413b1b0 con 0x7f78940a1960 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 >> v1:172.21.15.45:6790/0 conn(0x7f789409dd50 legacy=0x7f78941ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 >> v1:172.21.15.46:6789/0 conn(0x7f78940a2450 legacy=0x7f78941d54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 monclient: found mon.a 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789411fa20 con 0x7f78940a1960 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient: _finish_auth 0 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:04:54.877 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 5 monclient: authenticate success, global_id 4209 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient: _renew_subs 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78941366d0 con 0x7f78940a1960 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient: _renew_subs 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f789411d780 con 0x7f78940a1960 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 1 librados: init done 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78880047f0 con 0x7f78940a1960 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7888004b40 con 0x7f78940a1960 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:54.878 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7890d99640 4 set_mon_vals no callback set 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 monclient: dump: 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr: 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7888010bb0 con 0x7f78940a1960 2024-09-16T03:04:54.879 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7878005090 con 0x7f78940a1960 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 3606813768 0 0) 0x7f788804af20 con 0x7f78940a1960 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 client.4209.objecter ms_dispatch 0x7f7894000ba0 osd_map(15..15 src has 1..15) 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 3 client.4209.objecter handle_osd_map got epochs [15,15] > 0 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 3 client.4209.objecter handle_osd_map decoding full epoch 15 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 20 client.4209.objecter dump_active .. 0 homeless 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f789259c640 10 monclient: get_auth_request con 0x7f787403e260 auth_method 0 2024-09-16T03:04:54.880 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.859+0000 7f7882ffd640 10 client.4209.objecter ms_handle_connect 0x7f787403e260 2024-09-16T03:04:54.881 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.863+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f7888016a50 con 0x7f78940a1960 2024-09-16T03:04:54.881 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.863+0000 7f7882ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:54.881 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:54.863+0000 7f7882ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:55.040 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.035+0000 7f7898dac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:55.040 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.035+0000 7f7898dac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:55.040 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.035+0000 7f7898dac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:55.040 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.035+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7878005380 con 0x7f78940a1960 2024-09-16T03:04:55.041 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7882ffd640 1 -- 172.21.15.45:0/3325666228 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+9017 (unknown 4221056075 0 775497424) 0x7f7888016350 con 0x7f78940a1960 2024-09-16T03:04:55.041 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7882ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:04:55.041 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7882ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:55.041 INFO:teuthology.misc.health.smithi045.stdout: 2024-09-16T03:04:55.041 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":15,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","created":"2024-09-16T03:04:38.421437+0000","modified":"2024-09-16T03:04:51.559382+0000","last_up_change":"2024-09-16T03:04:48.525305+0000","last_in_change":"2024-09-16T03:04:40.127320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:04:49.504411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60a0a607-fe6c-4bd5-b369-e0dd6f6bd551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6800","nonce":3131549395}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6801","nonce":3131549395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6803","nonce":3131549395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6802","nonce":3131549395}]},"public_addr":"172.21.15.45:6800/3131549395","cluster_addr":"172.21.15.45:6801/3131549395","heartbeat_back_addr":"172.21.15.45:6803/3131549395","heartbeat_front_addr":"172.21.15.45:6802/3131549395","state":["exists","up"]},{"osd":1,"uuid":"10adca9c-3542-4901-a439-a1c9e9fcb8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6808","nonce":3228817795}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6809","nonce":3228817795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6811","nonce":3228817795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6810","nonce":3228817795}]},"public_addr":"172.21.15.45:6808/3228817795","cluster_addr":"172.21.15.45:6809/3228817795","heartbeat_back_addr":"172.21.15.45:6811/3228817795","heartbeat_front_addr":"172.21.15.45:6810/3228817795","state":["exists","up"]},{"osd":2,"uuid":"88fe1895-098a-4f13-95b2-4ead1739e71c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6804","nonce":207876738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6805","nonce":207876738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6807","nonce":207876738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6806","nonce":207876738}]},"public_addr":"172.21.15.45:6804/207876738","cluster_addr":"172.21.15.45:6805/207876738","heartbeat_back_addr":"172.21.15.45:6807/207876738","heartbeat_front_addr":"172.21.15.45:6806/207876738","state":["exists","up"]},{"osd":3,"uuid":"b04f94bc-5631-478e-904c-1dfc9a673a7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6808","nonce":3088806475}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6809","nonce":3088806475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6811","nonce":3088806475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6810","nonce":3088806475}]},"public_addr":"172.21.15.46:6808/3088806475","cluster_addr":"172.21.15.46:6809/3088806475","heartbeat_back_addr":"172.21.15.46:6811/3088806475","heartbeat_front_addr":"172.21.15.46:6810/3088806475","state":["exists","up"]},{"osd":4,"uuid":"c2dbca96-afb9-48ed-8a71-8a2bea37d152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6800","nonce":264199318}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":264199318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":264199318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":264199318}]},"public_addr":"172.21.15.46:6800/264199318","cluster_addr":"172.21.15.46:6801/264199318","heartbeat_back_addr":"172.21.15.46:6803/264199318","heartbeat_front_addr":"172.21.15.46:6802/264199318","state":["exists","up"]},{"osd":5,"uuid":"dabb8542-5498-4389-ad9e-0c0a54387723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":3390152579}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6805","nonce":3390152579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6807","nonce":3390152579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6806","nonce":3390152579}]},"public_addr":"172.21.15.46:6804/3390152579","cluster_addr":"172.21.15.46:6805/3390152579","heartbeat_back_addr":"172.21.15.46:6807/3390152579","heartbeat_front_addr":"172.21.15.46:6806/3390152579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.249472+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.780224+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.456157+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.830776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.656408+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.995384+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 10 librados: watch_flush enter 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 10 librados: watch_flush exit 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 20 client.4209.objecter shutdown clearing up homeless session... 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 10 client.4209.objecter successfully canceled tick 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 >> v1:172.21.15.46:6812/149360280 conn(0x7f787403e260 legacy=0x7f7874040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 10 monclient: shutdown 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 >> v1:172.21.15.45:6789/0 conn(0x7f78940a1960 legacy=0x7f78941d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 shutdown_connections 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 >> 172.21.15.45:0/3325666228 conn(0x7f78940a6580 msgr2=0x7f789413e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:55.044 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 shutdown_connections 2024-09-16T03:04:55.045 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 1 -- 172.21.15.45:0/3325666228 wait complete. 2024-09-16T03:04:55.045 INFO:teuthology.misc.health.smithi045.stderr:2024-09-16T03:04:55.039+0000 7f7898dac640 1 librados: shutdown 2024-09-16T03:04:55.057 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T03:04:55.058 INFO:tasks.ceph:Creating RBD pool 2024-09-16T03:04:55.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T03:04:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient: get_monmap_and_config 2024-09-16T03:04:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient: build_initial_monmap 2024-09-16T03:04:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient: monmap: 2024-09-16T03:04:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:55.173010+0000 2024-09-16T03:04:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:55.173010+0000 2024-09-16T03:04:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:55.175 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:55.175 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:55.175 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:55.175 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:55.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 1 Processor -- start 2024-09-16T03:04:55.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 1 -- start start 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient: init 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient(hunting): picked mon.noname-b con 0x7f0e34135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient(hunting): picked mon.noname-a con 0x7f0e3409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient(hunting): picked mon.noname-c con 0x7f0e3413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e3409f7e0 con 0x7f0e3409fc40 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e3409b160 con 0x7f0e3413c010 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e3409b440 con 0x7f0e34135950 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient(hunting): authenticate will time out at 985.822323s 2024-09-16T03:04:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3960d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0e34135950 0x7f0e34135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36096/0 (socket says 172.21.15.45:36096) 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e38e0c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0e3409fc40 0x7f0e3413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38722/0 (socket says 172.21.15.45:38722) 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3960d640 1 -- 172.21.15.45:0/3892783487 learned_addr learned my addr 172.21.15.45:0/3892783487 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1211402797 0 0) 0x7f0e3409f7e0 con 0x7f0e3409fc40 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient(hunting): my global_id is 4212 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e20003530 con 0x7f0e3409fc40 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 536766104 0 0) 0x7f0e3409b440 con 0x7f0e34135950 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient(hunting): my global_id is 4235 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e3409f7e0 con 0x7f0e34135950 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2448275015 0 0) 0x7f0e20003530 con 0x7f0e3409fc40 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 >> v1:172.21.15.45:6790/0 conn(0x7f0e34135950 legacy=0x7f0e34135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 >> v1:172.21.15.46:6789/0 conn(0x7f0e3413c010 legacy=0x7f0e3413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 monclient: found mon.noname-a 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e34136440 con 0x7f0e3409fc40 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: _finish_auth 0 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0e300015d0 con 0x7f0e3409fc40 2024-09-16T03:04:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: dump: 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 5 monclient: authenticate success, global_id 4212 2024-09-16T03:04:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e300017f0 con 0x7f0e3409fc40 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 1 -- 172.21.15.45:0/3892783487 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0e30001c40 con 0x7f0e3409fc40 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e2bfff640 10 monclient: dump: 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 4 set_mon_vals no callback set 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 10 monclient: shutdown 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.171+0000 7f0e3b871640 1 -- 172.21.15.45:0/3892783487 >> v1:172.21.15.45:6789/0 conn(0x7f0e3409fc40 legacy=0x7f0e3413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- 172.21.15.45:0/3892783487 shutdown_connections 2024-09-16T03:04:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- 172.21.15.45:0/3892783487 >> 172.21.15.45:0/3892783487 conn(0x7f0e340a2bd0 msgr2=0x7f0e340a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- 172.21.15.45:0/3892783487 shutdown_connections 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- 172.21.15.45:0/3892783487 wait complete. 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: build_initial_monmap 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: monmap: 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:55.190 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 librados: starting msgr at 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 librados: starting objecter 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 Processor -- start 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- start start 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 librados: setting wanted keys 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 librados: calling monclient init 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: init 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient(hunting): picked mon.noname-a con 0x7f0e3409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient(hunting): picked mon.noname-b con 0x7f0e34135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient(hunting): picked mon.noname-c con 0x7f0e3413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e34072760 con 0x7f0e3409fc40 2024-09-16T03:04:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e3413b6f0 con 0x7f0e34135950 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3960d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0e3409fc40 0x7f0e341cad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38724/0 (socket says 172.21.15.45:38724) 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3960d640 1 -- 172.21.15.45:0/1376328049 learned_addr learned my addr 172.21.15.45:0/1376328049 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e340714e0 con 0x7f0e3413c010 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient(hunting): authenticate will time out at 985.826034s 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 client.?.objecter ms_handle_connect 0x7f0e3409fc40 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 client.?.objecter ms_handle_connect 0x7f0e3413c010 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3701083523 0 0) 0x7f0e340714e0 con 0x7f0e3413c010 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:55.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): my global_id is 4238 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e18004ec0 con 0x7f0e3413c010 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 484980993 0 0) 0x7f0e3409f7e0 con 0x7f0e3409fc40 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): my global_id is 4215 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e340714e0 con 0x7f0e3409fc40 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 client.?.objecter ms_handle_connect 0x7f0e34135950 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3877889528 0 0) 0x7f0e18004ec0 con 0x7f0e3413c010 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e3409f7e0 con 0x7f0e3413c010 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0e2c001bc0 con 0x7f0e3413c010 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:55.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): dump: 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1161513456 0 0) 0x7f0e340714e0 con 0x7f0e3409fc40 2024-09-16T03:04:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e18004ec0 con 0x7f0e3409fc40 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0e24001940 con 0x7f0e3409fc40 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): dump: 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:55.195 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4175997594 0 0) 0x7f0e3413b6f0 con 0x7f0e34135950 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient(hunting): my global_id is 4231 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e340714e0 con 0x7f0e34135950 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1114863447 0 0) 0x7f0e3409f7e0 con 0x7f0e3413c010 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 >> v1:172.21.15.46:6789/0 conn(0x7f0e34135950 legacy=0x7f0e341cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 >> v1:172.21.15.45:6789/0 conn(0x7f0e3409fc40 legacy=0x7f0e341cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 monclient: found mon.c 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:55.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e340717c0 con 0x7f0e3413c010 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient: _finish_auth 0 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 5 monclient: authenticate success, global_id 4238 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: _renew_subs 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e34070580 con 0x7f0e3413c010 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: _renew_subs 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e3406f3c0 con 0x7f0e3413c010 2024-09-16T03:04:55.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 librados: init done 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e2c003980 con 0x7f0e3413c010 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0e2c003cd0 con 0x7f0e3413c010 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e2b7fe640 4 set_mon_vals no callback set 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 10 monclient: dump: 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f0e2c00fd40 con 0x7f0e3413c010 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.175+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e14005090 con 0x7f0e3413c010 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.179+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 3606813768 0 0) 0x7f0e2c049f00 con 0x7f0e3413c010 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.179+0000 7f0e29ffb640 10 client.4238.objecter ms_dispatch 0x7f0e34000ba0 osd_map(15..15 src has 1..15) 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.179+0000 7f0e29ffb640 3 client.4238.objecter handle_osd_map got epochs [15,15] > 0 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.179+0000 7f0e29ffb640 3 client.4238.objecter handle_osd_map decoding full epoch 15 2024-09-16T03:04:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.179+0000 7f0e29ffb640 20 client.4238.objecter dump_active .. 0 homeless 2024-09-16T03:04:55.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.179+0000 7f0e3960d640 10 monclient: get_auth_request con 0x7f0e1803e210 auth_method 0 2024-09-16T03:04:55.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.179+0000 7f0e29ffb640 10 client.4238.objecter ms_handle_connect 0x7f0e1803e210 2024-09-16T03:04:55.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.187+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f0e2c0153e0 con 0x7f0e3413c010 2024-09-16T03:04:55.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.187+0000 7f0e29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:55.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.187+0000 7f0e29ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.359+0000 7f0e3b871640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T03:04:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.359+0000 7f0e3b871640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T03:04:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.359+0000 7f0e3b871640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:55.359+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f0e14005c50 con 0x7f0e3413c010 2024-09-16T03:04:56.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.103+0000 7f0e29ffb640 1 -- 172.21.15.45:0/1376328049 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (unknown 231814880 0 0) 0x7f0e2c019db0 con 0x7f0e3413c010 2024-09-16T03:04:56.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.103+0000 7f0e29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T03:04:56.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.103+0000 7f0e29ffb640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T03:04:56.109 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'rbd' created 2024-09-16T03:04:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.107+0000 7f0e3b871640 10 librados: watch_flush enter 2024-09-16T03:04:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.107+0000 7f0e3b871640 10 librados: watch_flush exit 2024-09-16T03:04:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.107+0000 7f0e3b871640 20 client.4238.objecter shutdown clearing up homeless session... 2024-09-16T03:04:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.107+0000 7f0e3b871640 10 client.4238.objecter successfully canceled tick 2024-09-16T03:04:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.111+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 >> v1:172.21.15.46:6812/149360280 conn(0x7f0e1803e210 legacy=0x7f0e180406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.111+0000 7f0e3b871640 10 monclient: shutdown 2024-09-16T03:04:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.111+0000 7f0e3b871640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.111+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 >> v1:172.21.15.45:6790/0 conn(0x7f0e3413c010 legacy=0x7f0e341d3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.111+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 shutdown_connections 2024-09-16T03:04:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.111+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 >> 172.21.15.45:0/1376328049 conn(0x7f0e340a2bd0 msgr2=0x7f0e3413eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.111+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 shutdown_connections 2024-09-16T03:04:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.111+0000 7f0e3b871640 1 -- 172.21.15.45:0/1376328049 wait complete. 2024-09-16T03:04:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.111+0000 7f0e3b871640 1 librados: shutdown 2024-09-16T03:04:56.131 DEBUG:teuthology.orchestra.run.smithi045:> rbd --cluster ceph pool init rbd 2024-09-16T03:04:56.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient: get_monmap_and_config 2024-09-16T03:04:56.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient: build_initial_monmap 2024-09-16T03:04:56.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient: monmap: 2024-09-16T03:04:56.318 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:56.318 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:56.319 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:56.317673+0000 2024-09-16T03:04:56.319 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:56.317673+0000 2024-09-16T03:04:56.319 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:56.319 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:56.319 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:56.319 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:56.319 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:56.319 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:56.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 1 Processor -- start 2024-09-16T03:04:56.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 1 -- start start 2024-09-16T03:04:56.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient: init 2024-09-16T03:04:56.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient: _reopen_session rank -1 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient(hunting): picked mon.noname-b con 0x55f75ee6cae0 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient(hunting): picked mon.noname-c con 0x55f75edd7c40 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient(hunting): picked mon.noname-a con 0x55f75ee730d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55f75ec24c80 con 0x55f75ee730d0 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55f75edd7a00 con 0x55f75edd7c40 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55f75edbb6b0 con 0x55f75ee6cae0 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient(hunting): _renew_subs 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7e23f100 10 monclient(hunting): authenticate will time out at 986.967710s 2024-09-16T03:04:56.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7cd48640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x55f75ee6cae0 0x55f75ee6cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36102/0 (socket says 172.21.15.45:36102) 2024-09-16T03:04:56.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.315+0000 7fab7cd48640 1 -- 172.21.15.45:0/888573187 learned_addr learned my addr 172.21.15.45:0/888573187 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:56.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1154751613 0 0) 0x55f75edbb6b0 con 0x55f75ee6cae0 2024-09-16T03:04:56.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:56.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:56.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:56.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): my global_id is 4241 2024-09-16T03:04:56.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab60001e20 con 0x55f75ee6cae0 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1704066041 0 0) 0x55f75ec24c80 con 0x55f75ee730d0 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): my global_id is 4218 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55f75edbb6b0 con 0x55f75ee730d0 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3449468610 0 0) 0x55f75edd7a00 con 0x55f75edd7c40 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient(hunting): my global_id is 4234 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55f75ec24c80 con 0x55f75edd7c40 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3694499291 0 0) 0x55f75edbb6b0 con 0x55f75ee730d0 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 >> v1:172.21.15.45:6790/0 conn(0x55f75ee6cae0 legacy=0x55f75ee6cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 >> v1:172.21.15.46:6789/0 conn(0x55f75edd7c40 legacy=0x55f75ee719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: _finish_hunting 0 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 monclient: found mon.noname-a 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f75edbbf90 con 0x55f75ee730d0 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: _finish_auth 0 2024-09-16T03:04:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: _check_auth_tickets 2024-09-16T03:04:56.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:56.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fab70001a00 con 0x55f75ee730d0 2024-09-16T03:04:56.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:56.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:56.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: dump: 2024-09-16T03:04:56.331 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:56.332 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:56.332 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.332 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.332 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:56.332 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:56.333 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:56.333 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:56.333 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:56.333 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:56.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 5 monclient: authenticate success, global_id 4218 2024-09-16T03:04:56.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:56.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab70001ba0 con 0x55f75ee730d0 2024-09-16T03:04:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 1 -- 172.21.15.45:0/888573187 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fab70001f30 con 0x55f75ee730d0 2024-09-16T03:04:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab777fe640 10 monclient: dump: 2024-09-16T03:04:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.335 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:56.335 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:56.335 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:56.335 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:56.335 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:56.335 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:56.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 10 monclient: get_monmap_and_config success 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 4 set_mon_vals no callback set 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 10 monclient: shutdown 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 -- 172.21.15.45:0/888573187 >> v1:172.21.15.45:6789/0 conn(0x55f75ee730d0 legacy=0x55f75ee754c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 -- 172.21.15.45:0/888573187 shutdown_connections 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 -- 172.21.15.45:0/888573187 >> 172.21.15.45:0/888573187 conn(0x55f75eddb070 msgr2=0x55f75edbc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 -- 172.21.15.45:0/888573187 shutdown_connections 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 -- 172.21.15.45:0/888573187 wait complete. 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 10 monclient: build_initial_monmap 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 10 monclient: monmap: 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 librados: starting msgr at 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 librados: starting objecter 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 Processor -- start 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 -- start start 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 librados: setting wanted keys 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 1 librados: calling monclient init 2024-09-16T03:04:56.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 10 monclient: init 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 10 monclient: _reopen_session rank -1 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.319+0000 7fab7e23f100 10 monclient(hunting): picked mon.noname-c con 0x55f75ee6cae0 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient(hunting): picked mon.noname-a con 0x55f75ee730d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient(hunting): picked mon.noname-b con 0x55f75edd1150 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55f75ee6dcb0 con 0x55f75ee730d0 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55f75ee6df90 con 0x55f75edd1150 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55f75ee728a0 con 0x55f75ee6cae0 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient(hunting): _renew_subs 2024-09-16T03:04:56.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient(hunting): authenticate will time out at 986.972108s 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7cd48640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x55f75ee6cae0 0x55f75ee658c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36116/0 (socket says 172.21.15.45:36116) 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7cd48640 1 -- 172.21.15.45:0/1505277144 learned_addr learned my addr 172.21.15.45:0/1505277144 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 client.?.objecter ms_handle_connect 0x55f75ee6cae0 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 client.?.objecter ms_handle_connect 0x55f75ee730d0 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1653592565 0 0) 0x55f75ee728a0 con 0x55f75ee6cae0 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): my global_id is 4244 2024-09-16T03:04:56.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab5c002ff0 con 0x55f75ee6cae0 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4095091909 0 0) 0x55f75ee6dcb0 con 0x55f75ee730d0 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): my global_id is 4221 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55f75ee728a0 con 0x55f75ee730d0 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 client.?.objecter ms_handle_connect 0x55f75edd1150 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 261657680 0 0) 0x55f75ee6df90 con 0x55f75edd1150 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:56.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): my global_id is 4237 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55f75ee6dcb0 con 0x55f75edd1150 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3796994737 0 0) 0x55f75ee728a0 con 0x55f75ee730d0 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x55f75ee6df90 con 0x55f75ee730d0 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fab6c0018c0 con 0x55f75ee730d0 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): dump: 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3736713104 0 0) 0x7fab5c002ff0 con 0x55f75ee6cae0 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x55f75ee728a0 con 0x55f75ee6cae0 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fab680013c0 con 0x55f75ee6cae0 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:56.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient(hunting): dump: 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1144052932 0 0) 0x55f75ee6df90 con 0x55f75ee730d0 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 >> v1:172.21.15.45:6790/0 conn(0x55f75ee6cae0 legacy=0x55f75ee658c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 >> v1:172.21.15.46:6789/0 conn(0x55f75edd1150 legacy=0x55f75ee839a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 monclient: found mon.a 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f75ee6d9d0 con 0x55f75ee730d0 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient: _finish_auth 0 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 5 monclient: authenticate success, global_id 4221 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient: _renew_subs 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x55f75edd23f0 con 0x55f75ee730d0 2024-09-16T03:04:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab6c001f10 con 0x55f75ee730d0 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fab6c0022e0 con 0x55f75ee730d0 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient: _renew_subs 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x55f75ee852d0 con 0x55f75ee730d0 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 1 librados: init done 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 librados: wait_for_osdmap waiting 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 monclient: dump: 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab76ffd640 4 set_mon_vals no callback set 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fab6c002500 con 0x55f75ee730d0 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(16..16 src has 1..16) ==== 3495+0+0 (unknown 4235369303 0 0) 0x7fab6c048650 con 0x55f75ee730d0 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 osd_map(16..16 src has 1..16) 2024-09-16T03:04:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 3 client.4221.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T03:04:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 3 client.4221.objecter handle_osd_map decoding full epoch 16 2024-09-16T03:04:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 20 client.4221.objecter dump_active .. 0 homeless 2024-09-16T03:04:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 librados: wait_for_osdmap done waiting 2024-09-16T03:04:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T03:04:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T03:04:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55f75ee856b0 con 0x55f75ee730d0 2024-09-16T03:04:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab7cd48640 10 monclient: get_auth_request con 0x7fab5c03c3a0 auth_method 0 2024-09-16T03:04:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:56.323+0000 7fab74ff9640 10 client.4221.objecter ms_handle_connect 0x7fab5c03c3a0 2024-09-16T03:04:57.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.099+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (unknown 1851909647 0 0) 0x7fab6c0184f0 con 0x55f75ee730d0 2024-09-16T03:04:57.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.099+0000 7fab74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T03:04:57.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.099+0000 7fab74ff9640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T03:04:57.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.099+0000 7fab7e23f100 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:57.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.103+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fab70001ba0 con 0x55f75ee730d0 2024-09-16T03:04:57.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.107+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7fab6c0138c0 con 0x55f75ee730d0 2024-09-16T03:04:57.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.107+0000 7fab74ff9640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-16T03:04:57.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.107+0000 7fab76ffd640 10 client.4221.objecter _get_latest_version latest 17, waiting 2024-09-16T03:04:57.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.107+0000 7fab76ffd640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:57.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.107+0000 7fab76ffd640 10 monclient: _renew_subs 2024-09-16T03:04:57.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.107+0000 7fab76ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:57.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.107+0000 7fab76ffd640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=17}) -- 0x7fab54003ca0 con 0x55f75ee730d0 2024-09-16T03:04:57.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 11 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (unknown 1555451152 0 0) 0x7fab6c01cfa0 con 0x55f75ee730d0 2024-09-16T03:04:57.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab74ff9640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 osd_map(17..17 src has 1..17) 2024-09-16T03:04:57.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab74ff9640 3 client.4221.objecter handle_osd_map got epochs [17,17] > 16 2024-09-16T03:04:57.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab74ff9640 3 client.4221.objecter handle_osd_map decoding incremental epoch 17 2024-09-16T03:04:57.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab74ff9640 20 client.4221.objecter dump_active .. 0 homeless 2024-09-16T03:04:57.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab7e23f100 10 librados: create oid=rbd_trash nspace= 2024-09-16T03:04:57.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab7e23f100 10 client.4221.objecter _op_submit op 0x55f75ee85a60 2024-09-16T03:04:57.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab7e23f100 20 client.4221.objecter _calc_target epoch 17 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:04:57.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab7e23f100 20 client.4221.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-16T03:04:57.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.111+0000 7fab7e23f100 10 client.4221.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-09-16T03:04:57.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab7e23f100 20 client.4221.objecter _get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:57.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab7e23f100 10 client.4221.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-09-16T03:04:57.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab7e23f100 20 client.4221.objecter get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:57.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab7e23f100 15 client.4221.objecter _session_op_assign 5 1 2024-09-16T03:04:57.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab7e23f100 15 client.4221.objecter _send_op 1 to 2.3 on osd.5 2024-09-16T03:04:57.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.46:6804/3390152579 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x55f75ee8a0a0 con 0x55f75ee86590 2024-09-16T03:04:57.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab7e23f100 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:57.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab7e23f100 5 client.4221.objecter 1 in flight 2024-09-16T03:04:57.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab7dfaa640 10 monclient: get_auth_request con 0x55f75ee86590 auth_method 0 2024-09-16T03:04:57.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.115+0000 7fab74ff9640 10 client.4221.objecter ms_handle_connect 0x55f75ee86590 2024-09-16T03:04:57.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 1 -- 172.21.15.45:0/1505277144 <== osd.5 v1:172.21.15.46:6804/3390152579 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (unknown 1703302865 0 0) 0x7fab70001720 con 0x55f75ee86590 2024-09-16T03:04:57.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) 2024-09-16T03:04:57.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 10 client.4221.objecter in handle_osd_op_reply 2024-09-16T03:04:57.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 7 client.4221.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-16T03:04:57.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 10 client.4221.objecter op 0 rval 0 len 0 2024-09-16T03:04:57.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 15 client.4221.objecter handle_osd_op_reply completed tid 1 2024-09-16T03:04:57.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 15 client.4221.objecter _finish_op 1 2024-09-16T03:04:57.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:57.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 15 client.4221.objecter _session_op_remove 5 1 2024-09-16T03:04:57.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7dfaa640 5 client.4221.objecter 0 in flight 2024-09-16T03:04:57.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.119+0000 7fab7e23f100 10 librados: Objecter returned from create r=0 2024-09-16T03:04:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 10 client.4221.objecter _op_submit op 0x55f75eedc4e0 2024-09-16T03:04:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 20 client.4221.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 20 client.4221.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 10 client.4221.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 20 client.4221.objecter _get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 10 client.4221.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 20 client.4221.objecter get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 15 client.4221.objecter _session_op_assign 5 2 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 15 client.4221.objecter _send_op 2 to 2.2 on osd.5 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.46:6804/3390152579 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55f75eedca30 con 0x55f75ee86590 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7e23f100 5 client.4221.objecter 1 in flight 2024-09-16T03:04:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 1 -- 172.21.15.45:0/1505277144 <== osd.5 v1:172.21.15.46:6804/3390152579 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4274215525 0 0) 0x7fab70001720 con 0x55f75ee86590 2024-09-16T03:04:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:04:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 10 client.4221.objecter in handle_osd_op_reply 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 7 client.4221.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 10 client.4221.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 10 client.4221.objecter op 0 rval 0 len 0 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 15 client.4221.objecter handle_osd_op_reply completed tid 2 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 15 client.4221.objecter _finish_op 2 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 15 client.4221.objecter _session_op_remove 5 2 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.123+0000 7fab7dfaa640 5 client.4221.objecter 0 in flight 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 10 client.4221.objecter _op_submit op 0x55f75eedb670 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 20 client.4221.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 20 client.4221.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 10 client.4221.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:04:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 20 client.4221.objecter _get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 10 client.4221.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 20 client.4221.objecter get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 15 client.4221.objecter _session_op_assign 5 3 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 15 client.4221.objecter _send_op 3 to 2.2 on osd.5 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.46:6804/3390152579 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55f75eedbb20 con 0x55f75ee86590 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7e23f100 5 client.4221.objecter 1 in flight 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 1 -- 172.21.15.45:0/1505277144 <== osd.5 v1:172.21.15.46:6804/3390152579 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4287864937 0 0) 0x7fab70001720 con 0x55f75ee86590 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 10 client.4221.objecter in handle_osd_op_reply 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 7 client.4221.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 10 client.4221.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 10 client.4221.objecter op 0 rval 0 len 0 2024-09-16T03:04:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 15 client.4221.objecter handle_osd_op_reply completed tid 3 2024-09-16T03:04:57.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 15 client.4221.objecter _finish_op 3 2024-09-16T03:04:57.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:57.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 15 client.4221.objecter _session_op_remove 5 3 2024-09-16T03:04:57.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab7dfaa640 5 client.4221.objecter 0 in flight 2024-09-16T03:04:57.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab767fc640 10 client.4221.objecter allocate_selfmanaged_snap; pool: 2 2024-09-16T03:04:57.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab767fc640 10 client.4221.objecter pool_op_submit 4 2024-09-16T03:04:57.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab767fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:57.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:57.127+0000 7fab767fc640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7fab580030e0 con 0x55f75ee730d0 2024-09-16T03:04:58.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 12 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (unknown 3595471822 0 0) 0x7fab6c01d430 con 0x55f75ee730d0 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 pool_op_reply(tid 4 (0) Success v18) 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 10 client.4221.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v18) 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 10 client.4221.objecter have request 4 at 0x7fab58003070 Op: create unmanaged snap 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 20 client.4221.objecter waiting for client to reach epoch 18 before calling back 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 10 monclient: _renew_subs 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=18}) -- 0x55f75ee6df90 con 0x55f75ee730d0 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 10 client.4221.objecter done 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 13 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (unknown 2940631289 0 0) 0x7fab6c01d1a0 con 0x55f75ee730d0 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 osd_map(18..18 src has 1..18) 2024-09-16T03:04:58.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 3 client.4221.objecter handle_osd_map got epochs [18,18] > 17 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 3 client.4221.objecter handle_osd_map decoding incremental epoch 18 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab74ff9640 20 client.4221.objecter dump_active .. 0 homeless 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 20 librados: queue_aio_write 0x55f75eed7a70 completion 0x55f75eca5ba0 write_seq 1 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 10 client.4221.objecter _op_submit op 0x7fab54006390 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 20 client.4221.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 20 client.4221.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 10 client.4221.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 20 client.4221.objecter _get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 10 client.4221.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 20 client.4221.objecter get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 15 client.4221.objecter _session_op_assign 5 5 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 15 client.4221.objecter _send_op 5 to 2.2 on osd.5 2024-09-16T03:04:58.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.46:6804/3390152579 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7fab540069d0 con 0x55f75ee86590 2024-09-16T03:04:58.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:58.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.123+0000 7fab76ffd640 5 client.4221.objecter 1 in flight 2024-09-16T03:04:58.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 1 -- 172.21.15.45:0/1505277144 <== osd.5 v1:172.21.15.46:6804/3390152579 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (unknown 2910594948 0 0) 0x7fab70001720 con 0x55f75ee86590 2024-09-16T03:04:58.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) 2024-09-16T03:04:58.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 10 client.4221.objecter in handle_osd_op_reply 2024-09-16T03:04:58.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 7 client.4221.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-16T03:04:58.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 10 client.4221.objecter op 0 rval 0 len 0 2024-09-16T03:04:58.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 10 client.4221.objecter op 1 rval 0 len 0 2024-09-16T03:04:58.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 15 client.4221.objecter handle_osd_op_reply completed tid 5 2024-09-16T03:04:58.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 15 client.4221.objecter _finish_op 5 2024-09-16T03:04:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 15 client.4221.objecter _session_op_remove 5 5 2024-09-16T03:04:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 5 client.4221.objecter 0 in flight 2024-09-16T03:04:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab7dfaa640 20 librados: complete_aio_write 0x55f75eca5ba0 2024-09-16T03:04:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab767fc640 10 client.4221.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-16T03:04:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab767fc640 10 client.4221.objecter pool_op_submit 6 2024-09-16T03:04:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab767fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:58.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:58.131+0000 7fab767fc640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7fab58004550 con 0x55f75ee730d0 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.127+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 14 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (unknown 3262413325 0 0) 0x7fab6c013030 con 0x55f75ee730d0 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.127+0000 7fab74ff9640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 pool_op_reply(tid 6 (0) Success v19) 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.127+0000 7fab74ff9640 10 client.4221.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v19) 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.127+0000 7fab74ff9640 10 client.4221.objecter have request 6 at 0x7fab58003070 Op: delete unmanaged snap 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 20 client.4221.objecter waiting for client to reach epoch 19 before calling back 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 10 monclient: _renew_subs 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=19}) -- 0x7fab5c002ff0 con 0x55f75ee730d0 2024-09-16T03:04:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 10 client.4221.objecter done 2024-09-16T03:04:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 1 -- 172.21.15.45:0/1505277144 <== mon.0 v1:172.21.15.45:6789/0 15 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (unknown 1888694435 0 0) 0x7fab6c012ab0 con 0x55f75ee730d0 2024-09-16T03:04:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 osd_map(19..19 src has 1..19) 2024-09-16T03:04:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 3 client.4221.objecter handle_osd_map got epochs [19,19] > 18 2024-09-16T03:04:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 3 client.4221.objecter handle_osd_map decoding incremental epoch 19 2024-09-16T03:04:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab74ff9640 20 client.4221.objecter dump_active .. 0 homeless 2024-09-16T03:04:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 20 librados: queue_aio_write 0x55f75eed7a70 completion 0x7fab58001d70 write_seq 2 2024-09-16T03:04:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 10 client.4221.objecter _op_submit op 0x7fab54006390 2024-09-16T03:04:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 20 client.4221.objecter _calc_target epoch 19 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:04:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 20 client.4221.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:04:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 10 client.4221.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:04:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 20 client.4221.objecter _get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 10 client.4221.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-09-16T03:04:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 20 client.4221.objecter get_session s=0x55f75ee85f10 osd=5 3 2024-09-16T03:04:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 15 client.4221.objecter _session_op_assign 5 7 2024-09-16T03:04:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 15 client.4221.objecter _send_op 7 to 2.2 on osd.5 2024-09-16T03:04:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 1 -- 172.21.15.45:0/1505277144 --> v1:172.21.15.46:6804/3390152579 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7fab54006db0 con 0x55f75ee86590 2024-09-16T03:04:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.131+0000 7fab76ffd640 5 client.4221.objecter 1 in flight 2024-09-16T03:04:59.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 1 -- 172.21.15.45:0/1505277144 <== osd.5 v1:172.21.15.46:6804/3390152579 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (unknown 900577457 0 0) 0x7fab70001720 con 0x55f75ee86590 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 10 client.4221.objecter ms_dispatch 0x55f75ecbea30 osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 10 client.4221.objecter in handle_osd_op_reply 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 7 client.4221.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 10 client.4221.objecter op 0 rval 0 len 0 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 15 client.4221.objecter handle_osd_op_reply completed tid 7 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 15 client.4221.objecter _finish_op 7 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 4 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 15 client.4221.objecter _session_op_remove 5 7 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 5 client.4221.objecter 0 in flight 2024-09-16T03:04:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 20 librados: complete_aio_write 0x7fab58001d70 2024-09-16T03:04:59.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 10 librados: watch_flush enter 2024-09-16T03:04:59.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 10 librados: watch_flush exit 2024-09-16T03:04:59.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 10 client.4221.objecter close_session for osd.5 2024-09-16T03:04:59.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 >> v1:172.21.15.46:6804/3390152579 conn(0x55f75ee86590 legacy=0x55f75ee88980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 20 client.4221.objecter put_session s=0x55f75ee85f10 osd=5 1 2024-09-16T03:04:59.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 20 client.4221.objecter shutdown clearing up homeless session... 2024-09-16T03:04:59.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 10 client.4221.objecter successfully canceled tick 2024-09-16T03:04:59.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 >> v1:172.21.15.46:6812/149360280 conn(0x7fab5c03c3a0 legacy=0x7fab5c03e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 10 monclient: shutdown 2024-09-16T03:04:59.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:59.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 >> v1:172.21.15.45:6789/0 conn(0x55f75ee730d0 legacy=0x55f75ee65fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7dfaa640 1 -- 172.21.15.45:0/1505277144 reap_dead start 2024-09-16T03:04:59.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 shutdown_connections 2024-09-16T03:04:59.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 >> 172.21.15.45:0/1505277144 conn(0x55f75eddb070 msgr2=0x55f75eddcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:59.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 shutdown_connections 2024-09-16T03:04:59.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 1 -- 172.21.15.45:0/1505277144 wait complete. 2024-09-16T03:04:59.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.139+0000 7fab7e23f100 1 librados: shutdown 2024-09-16T03:04:59.148 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T03:04:59.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-16T03:04:59.149 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T03:04:59.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient: get_monmap_and_config 2024-09-16T03:04:59.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient: build_initial_monmap 2024-09-16T03:04:59.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient: monmap: 2024-09-16T03:04:59.278 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:59.278 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.278 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:59.277575+0000 2024-09-16T03:04:59.279 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:59.277575+0000 2024-09-16T03:04:59.279 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:59.279 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.279 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:59.279 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:59.279 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:59.279 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 1 Processor -- start 2024-09-16T03:04:59.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 1 -- start start 2024-09-16T03:04:59.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient: init 2024-09-16T03:04:59.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:59.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:04:59.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient(hunting): picked mon.noname-a con 0x7fe17c135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:59.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient(hunting): picked mon.noname-b con 0x7fe17c09fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:59.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient(hunting): picked mon.noname-c con 0x7fe17c13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c09f7e0 con 0x7fe17c135950 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c09b160 con 0x7fe17c13c010 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c09b440 con 0x7fe17c09fc40 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe183081640 10 monclient(hunting): authenticate will time out at 989.927203s 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe180e1d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe17c135950 0x7fe17c135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38750/0 (socket says 172.21.15.45:38750) 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe180e1d640 1 -- 172.21.15.45:0/3736162345 learned_addr learned my addr 172.21.15.45:0/3736162345 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2046908724 0 0) 0x7fe17c09f7e0 con 0x7fe17c135950 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 10 monclient(hunting): my global_id is 4224 2024-09-16T03:04:59.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe168003530 con 0x7fe17c135950 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 563812368 0 0) 0x7fe17c09b440 con 0x7fe17c09fc40 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 10 monclient(hunting): my global_id is 4247 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.275+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe17c09f7e0 con 0x7fe17c09fc40 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4021738888 0 0) 0x7fe17c09b160 con 0x7fe17c13c010 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient(hunting): my global_id is 4240 2024-09-16T03:04:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe17c09b440 con 0x7fe17c13c010 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2982510392 0 0) 0x7fe168003530 con 0x7fe17c135950 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 >> v1:172.21.15.45:6790/0 conn(0x7fe17c09fc40 legacy=0x7fe17c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 >> v1:172.21.15.46:6789/0 conn(0x7fe17c13c010 legacy=0x7fe17c13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c136440 con 0x7fe17c135950 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe1740016a0 con 0x7fe17c135950 2024-09-16T03:04:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: dump: 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 5 monclient: authenticate success, global_id 4224 2024-09-16T03:04:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe174001840 con 0x7fe17c135950 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 1 -- 172.21.15.45:0/3736162345 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe174001cd0 con 0x7fe17c135950 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 4 set_mon_vals no callback set 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17b7fe640 10 monclient: dump: 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: shutdown 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- 172.21.15.45:0/3736162345 >> v1:172.21.15.45:6789/0 conn(0x7fe17c135950 legacy=0x7fe17c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- 172.21.15.45:0/3736162345 shutdown_connections 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- 172.21.15.45:0/3736162345 >> 172.21.15.45:0/3736162345 conn(0x7fe17c0a2bd0 msgr2=0x7fe17c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- 172.21.15.45:0/3736162345 shutdown_connections 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- 172.21.15.45:0/3736162345 wait complete. 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: build_initial_monmap 2024-09-16T03:04:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: monmap: 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 librados: starting msgr at 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 librados: starting objecter 2024-09-16T03:04:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 Processor -- start 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- start start 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 librados: setting wanted keys 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 librados: calling monclient init 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: init 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient(hunting): picked mon.noname-b con 0x7fe17c09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient(hunting): picked mon.noname-a con 0x7fe17c135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient(hunting): picked mon.noname-c con 0x7fe17c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c09d2c0 con 0x7fe17c135950 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c13b6f0 con 0x7fe17c09fc40 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c09e770 con 0x7fe17c13c010 2024-09-16T03:04:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient(hunting): authenticate will time out at 989.930782s 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe17c135950 0x7fe17c1d1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38762/0 (socket says 172.21.15.45:38762) 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe17bfff640 1 -- 172.21.15.45:0/905880140 learned_addr learned my addr 172.21.15.45:0/905880140 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 client.?.objecter ms_handle_connect 0x7fe17c135950 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 client.?.objecter ms_handle_connect 0x7fe17c13c010 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3216819998 0 0) 0x7fe17c09d2c0 con 0x7fe17c135950 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): my global_id is 4227 2024-09-16T03:04:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe15c004ec0 con 0x7fe17c135950 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1202460589 0 0) 0x7fe17c09e770 con 0x7fe17c13c010 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): my global_id is 4250 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe17c09d2c0 con 0x7fe17c13c010 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 client.?.objecter ms_handle_connect 0x7fe17c09fc40 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1023569370 0 0) 0x7fe17c09d2c0 con 0x7fe17c13c010 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe17c09e770 con 0x7fe17c13c010 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe16c001900 con 0x7fe17c13c010 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): dump: 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258840419 0 0) 0x7fe17c13b6f0 con 0x7fe17c09fc40 2024-09-16T03:04:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): my global_id is 4243 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe17c09d2c0 con 0x7fe17c09fc40 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3400900486 0 0) 0x7fe15c004ec0 con 0x7fe17c135950 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe17c13b6f0 con 0x7fe17c135950 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe164001ce0 con 0x7fe17c135950 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient(hunting): dump: 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1918192209 0 0) 0x7fe17c09e770 con 0x7fe17c13c010 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 >> v1:172.21.15.46:6789/0 conn(0x7fe17c09fc40 legacy=0x7fe17c1ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 >> v1:172.21.15.45:6789/0 conn(0x7fe17c135950 legacy=0x7fe17c1d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient: _finish_hunting 0 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 monclient: found mon.c 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c09ea50 con 0x7fe17c13c010 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient: _finish_auth 0 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 10 monclient: _check_auth_tickets 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe1797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 5 monclient: authenticate success, global_id 4250 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: _renew_subs 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c09ed30 con 0x7fe17c13c010 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: _renew_subs 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe17c0739a0 con 0x7fe17c13c010 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.279+0000 7fe183081640 1 librados: init done 2024-09-16T03:04:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe16c003620 con 0x7fe17c13c010 2024-09-16T03:04:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe16c0039b0 con 0x7fe17c13c010 2024-09-16T03:04:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe17a7fc640 4 set_mon_vals no callback set 2024-09-16T03:04:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 10 monclient: dump: 2024-09-16T03:04:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe183081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe183081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe183081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe160005090 con 0x7fe17c13c010 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fe16c003bd0 con 0x7fe17c13c010 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fe16c049d90 con 0x7fe17c13c010 2024-09-16T03:04:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 10 client.4250.objecter ms_dispatch 0x7fe17c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:04:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 3 client.4250.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:04:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 3 client.4250.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:04:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 20 client.4250.objecter dump_active .. 0 homeless 2024-09-16T03:04:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe180e1d640 10 monclient: get_auth_request con 0x7fe15c03e2b0 auth_method 0 2024-09-16T03:04:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.283+0000 7fe1797fa640 10 client.4250.objecter ms_handle_connect 0x7fe15c03e2b0 2024-09-16T03:04:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.287+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fe16c019af0 con 0x7fe17c13c010 2024-09-16T03:04:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.287+0000 7fe1797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.287+0000 7fe1797fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:59.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.458+0000 7fe183081640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-16T03:04:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.458+0000 7fe183081640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-16T03:04:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.458+0000 7fe183081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.458+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fe160005380 con 0x7fe17c13c010 2024-09-16T03:04:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.458+0000 7fe1797fa640 1 -- 172.21.15.45:0/905880140 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v6) ==== 86+0+159 (unknown 2698850854 0 605716991) 0x7fe16c01ea30 con 0x7fe17c13c010 2024-09-16T03:04:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.458+0000 7fe1797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-16T03:04:59.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.458+0000 7fe1797fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:59.464 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 10 librados: watch_flush enter 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 10 librados: watch_flush exit 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 20 client.4250.objecter shutdown clearing up homeless session... 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 10 client.4250.objecter successfully canceled tick 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 >> v1:172.21.15.46:6812/149360280 conn(0x7fe15c03e2b0 legacy=0x7fe15c040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 10 monclient: shutdown 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 >> v1:172.21.15.45:6790/0 conn(0x7fe17c13c010 legacy=0x7fe17c1d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 shutdown_connections 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 >> 172.21.15.45:0/905880140 conn(0x7fe17c0a2bd0 msgr2=0x7fe17c13ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 shutdown_connections 2024-09-16T03:04:59.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 1 -- 172.21.15.45:0/905880140 wait complete. 2024-09-16T03:04:59.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.462+0000 7fe183081640 1 librados: shutdown 2024-09-16T03:04:59.482 INFO:teuthology.orchestra.run.smithi045.stdout:[{"version":6,"timestamp":"2024-09-16T03:04:48.434084+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"7257.815040"}]}] 2024-09-16T03:04:59.483 INFO:tasks.ceph_manager:config epoch is 6 2024-09-16T03:04:59.483 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T03:04:59.483 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T03:04:59.483 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T03:04:59.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.602+0000 7fe52578e640 10 monclient: get_monmap_and_config 2024-09-16T03:04:59.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.602+0000 7fe52578e640 10 monclient: build_initial_monmap 2024-09-16T03:04:59.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.602+0000 7fe52578e640 10 monclient: monmap: 2024-09-16T03:04:59.607 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:59.607 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.607 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:59.606811+0000 2024-09-16T03:04:59.608 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:59.606811+0000 2024-09-16T03:04:59.608 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:59.608 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.608 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:59.608 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:59.608 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:59.608 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.602+0000 7fe52578e640 1 Processor -- start 2024-09-16T03:04:59.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.602+0000 7fe52578e640 1 -- start start 2024-09-16T03:04:59.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.602+0000 7fe52578e640 10 monclient: init 2024-09-16T03:04:59.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:59.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:04:59.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): picked mon.noname-c con 0x7fe5200a0ff0 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:59.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): picked mon.noname-b con 0x7fe5200a18b0 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): picked mon.noname-a con 0x7fe52013c720 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe52009d810 con 0x7fe52013c720 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5200746f0 con 0x7fe5200a0ff0 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe52009ec80 con 0x7fe5200a18b0 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): authenticate will time out at 990.256409s 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe52013c720 0x7fe52013eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38776/0 (socket says 172.21.15.45:38776) 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51e7fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe5200a18b0 0x7fe52009d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36148/0 (socket says 172.21.15.45:36148) 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51f7fe640 1 -- 172.21.15.45:0/3959005166 learned_addr learned my addr 172.21.15.45:0/3959005166 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263634648 0 0) 0x7fe52009ec80 con 0x7fe5200a18b0 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): my global_id is 4253 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe508003530 con 0x7fe5200a18b0 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 971903363 0 0) 0x7fe52009d810 con 0x7fe52013c720 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): my global_id is 4230 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe52009ec80 con 0x7fe52013c720 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1564793311 0 0) 0x7fe5200746f0 con 0x7fe5200a0ff0 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient(hunting): my global_id is 4246 2024-09-16T03:04:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe52009d810 con 0x7fe5200a0ff0 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 352106452 0 0) 0x7fe508003530 con 0x7fe5200a18b0 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 >> v1:172.21.15.46:6789/0 conn(0x7fe5200a0ff0 legacy=0x7fe52009c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 >> v1:172.21.15.45:6789/0 conn(0x7fe52013c720 legacy=0x7fe52013eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 monclient: found mon.noname-b 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52009f5b0 con 0x7fe5200a18b0 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: _finish_auth 0 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe5140015d0 con 0x7fe5200a18b0 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 5 monclient: authenticate success, global_id 4253 2024-09-16T03:04:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: dump: 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5140017f0 con 0x7fe5200a18b0 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 1 -- 172.21.15.45:0/3959005166 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe514001bc0 con 0x7fe5200a18b0 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 4 set_mon_vals no callback set 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51dffb640 10 monclient: dump: 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.616 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient: shutdown 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- 172.21.15.45:0/3959005166 >> v1:172.21.15.45:6790/0 conn(0x7fe5200a18b0 legacy=0x7fe52009d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- 172.21.15.45:0/3959005166 shutdown_connections 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- 172.21.15.45:0/3959005166 >> 172.21.15.45:0/3959005166 conn(0x7fe5200a4a10 msgr2=0x7fe5200a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- 172.21.15.45:0/3959005166 shutdown_connections 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- 172.21.15.45:0/3959005166 wait complete. 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient: build_initial_monmap 2024-09-16T03:04:59.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient: monmap: 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 librados: starting msgr at 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 librados: starting objecter 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 Processor -- start 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- start start 2024-09-16T03:04:59.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 librados: setting wanted keys 2024-09-16T03:04:59.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 librados: calling monclient init 2024-09-16T03:04:59.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient: init 2024-09-16T03:04:59.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:59.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:04:59.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): picked mon.noname-a con 0x7fe5200a0ff0 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:59.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): picked mon.noname-b con 0x7fe5200a18b0 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:59.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): picked mon.noname-c con 0x7fe52013c720 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:59.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe520126e30 con 0x7fe5200a0ff0 2024-09-16T03:04:59.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe52009f100 con 0x7fe5200a18b0 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5201288f0 con 0x7fe52013c720 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe52578e640 10 monclient(hunting): authenticate will time out at 990.259514s 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe5200a0ff0 0x7fe52013b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38784/0 (socket says 172.21.15.45:38784) 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe51effd640 1 -- 172.21.15.45:0/1584829522 learned_addr learned my addr 172.21.15.45:0/1584829522 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 client.?.objecter ms_handle_connect 0x7fe52013c720 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 client.?.objecter ms_handle_connect 0x7fe5200a0ff0 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370308439 0 0) 0x7fe5201288f0 con 0x7fe52013c720 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 monclient(hunting): my global_id is 4256 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe50c004ec0 con 0x7fe52013c720 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 client.?.objecter ms_handle_connect 0x7fe5200a18b0 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940487818 0 0) 0x7fe520126e30 con 0x7fe5200a0ff0 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 10 monclient(hunting): my global_id is 4233 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.606+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5201288f0 con 0x7fe5200a0ff0 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1828034718 0 0) 0x7fe52009f100 con 0x7fe5200a18b0 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): my global_id is 4249 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe520126e30 con 0x7fe5200a18b0 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3659396174 0 0) 0x7fe5201288f0 con 0x7fe5200a0ff0 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe52009f100 con 0x7fe5200a0ff0 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe510001a00 con 0x7fe5200a0ff0 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): dump: 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.625 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2960364160 0 0) 0x7fe50c004ec0 con 0x7fe52013c720 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5201288f0 con 0x7fe52013c720 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe518001c40 con 0x7fe52013c720 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient(hunting): dump: 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1118528432 0 0) 0x7fe52009f100 con 0x7fe5200a0ff0 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 >> v1:172.21.15.45:6790/0 conn(0x7fe52013c720 legacy=0x7fe5201d5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 >> v1:172.21.15.46:6789/0 conn(0x7fe5200a18b0 legacy=0x7fe5201d1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient: _finish_hunting 0 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 monclient: found mon.a 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe520128bd0 con 0x7fe5200a0ff0 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient: _finish_auth 0 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient: _check_auth_tickets 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 5 monclient: authenticate success, global_id 4233 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 10 monclient: _renew_subs 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe52013bee0 con 0x7fe5200a0ff0 2024-09-16T03:04:59.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:59.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 10 monclient: _renew_subs 2024-09-16T03:04:59.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:59.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5201ccdd0 con 0x7fe5200a0ff0 2024-09-16T03:04:59.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 1 librados: init done 2024-09-16T03:04:59.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe510001ca0 con 0x7fe5200a0ff0 2024-09-16T03:04:59.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:59.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe510003a70 con 0x7fe5200a0ff0 2024-09-16T03:04:59.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:59.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:59.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe51d7fa640 4 set_mon_vals no callback set 2024-09-16T03:04:59.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 monclient: dump: 2024-09-16T03:04:59.630 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.630 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.630 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.630 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe500005090 con 0x7fe5200a0ff0 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fe510003c10 con 0x7fe5200a0ff0 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fe51004b0e0 con 0x7fe5200a0ff0 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 client.4233.objecter ms_dispatch 0x7fe520000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:04:59.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 3 client.4233.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:04:59.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 3 client.4233.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:04:59.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 20 client.4233.objecter dump_active .. 0 homeless 2024-09-16T03:04:59.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe51e7fc640 10 monclient: get_auth_request con 0x7fe50c03e200 auth_method 0 2024-09-16T03:04:59.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.610+0000 7fe5077fe640 10 client.4233.objecter ms_handle_connect 0x7fe50c03e200 2024-09-16T03:04:59.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.614+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fe510016a20 con 0x7fe5200a0ff0 2024-09-16T03:04:59.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.614+0000 7fe5077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:59.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.614+0000 7fe5077fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:04:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.826+0000 7fe52578e640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T03:04:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.826+0000 7fe52578e640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T03:04:59.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.826+0000 7fe52578e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:04:59.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.826+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fe500005380 con 0x7fe5200a0ff0 2024-09-16T03:04:59.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.826+0000 7fe5077fe640 1 -- 172.21.15.45:0/1584829522 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+93709 (unknown 481731318 0 4151120173) 0x7fe51001ae00 con 0x7fe5200a0ff0 2024-09-16T03:04:59.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.826+0000 7fe5077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T03:04:59.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.826+0000 7fe5077fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:04:59.833 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 10 librados: watch_flush enter 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 10 librados: watch_flush exit 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 20 client.4233.objecter shutdown clearing up homeless session... 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 10 client.4233.objecter successfully canceled tick 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 >> v1:172.21.15.46:6812/149360280 conn(0x7fe50c03e200 legacy=0x7fe50c0406c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 10 monclient: shutdown 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 >> v1:172.21.15.45:6789/0 conn(0x7fe5200a0ff0 legacy=0x7fe52013b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 shutdown_connections 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 >> 172.21.15.45:0/1584829522 conn(0x7fe5200a4a10 msgr2=0x7fe52006cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 shutdown_connections 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 1 -- 172.21.15.45:0/1584829522 wait complete. 2024-09-16T03:04:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.830+0000 7fe52578e640 1 librados: shutdown 2024-09-16T03:04:59.849 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6812","nonce":149360280}]},"active_addr":"172.21.15.46:6812/149360280","active_change":"2024-09-16T03:04:43.447255+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":185107144}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":4167444082}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":2232155433}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3251060609}]}]} 2024-09-16T03:04:59.851 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T03:04:59.851 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T03:04:59.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:04:59.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.958+0000 7f6e0863b640 10 monclient: get_monmap_and_config 2024-09-16T03:04:59.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.958+0000 7f6e0863b640 10 monclient: build_initial_monmap 2024-09-16T03:04:59.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.958+0000 7f6e0863b640 10 monclient: monmap: 2024-09-16T03:04:59.963 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:59.962877+0000 2024-09-16T03:04:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:59.962877+0000 2024-09-16T03:04:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:04:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:04:59.964 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.958+0000 7f6e0863b640 1 Processor -- start 2024-09-16T03:04:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- start start 2024-09-16T03:04:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: init 2024-09-16T03:04:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): picked mon.noname-b con 0x7f6e00135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): picked mon.noname-a con 0x7f6e0009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): picked mon.noname-c con 0x7f6e0013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e0009f7e0 con 0x7f6e0009fc40 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e0009b160 con 0x7f6e0013c010 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e0009b440 con 0x7f6e00135950 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): authenticate will time out at 990.612733s 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e063d7640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6e00135950 0x7f6e00135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36156/0 (socket says 172.21.15.45:36156) 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e063d7640 1 -- 172.21.15.45:0/2499883883 learned_addr learned my addr 172.21.15.45:0/2499883883 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2978690841 0 0) 0x7f6e0009b440 con 0x7f6e00135950 2024-09-16T03:04:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient(hunting): my global_id is 4259 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dec003530 con 0x7f6e00135950 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1654576529 0 0) 0x7f6e0009f7e0 con 0x7f6e0009fc40 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient(hunting): my global_id is 4236 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e0009b440 con 0x7f6e0009fc40 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1422437696 0 0) 0x7f6dec003530 con 0x7f6e00135950 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 >> v1:172.21.15.46:6789/0 conn(0x7f6e0013c010 legacy=0x7f6e0013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 >> v1:172.21.15.45:6789/0 conn(0x7f6e0009fc40 legacy=0x7f6e0013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: _finish_hunting 0 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 monclient: found mon.noname-b 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e00136440 con 0x7f6e00135950 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: _finish_auth 0 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: _check_auth_tickets 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6df40016a0 con 0x7f6e00135950 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: dump: 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 5 monclient: authenticate success, global_id 4259 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6df40018c0 con 0x7f6e00135950 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: get_monmap_and_config success 2024-09-16T03:04:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 4 set_mon_vals no callback set 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 1 -- 172.21.15.45:0/2499883883 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6df4001d10 con 0x7f6e00135950 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e053d5640 10 monclient: dump: 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.972 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: shutdown 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- 172.21.15.45:0/2499883883 >> v1:172.21.15.45:6790/0 conn(0x7f6e00135950 legacy=0x7f6e00135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- 172.21.15.45:0/2499883883 shutdown_connections 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- 172.21.15.45:0/2499883883 >> 172.21.15.45:0/2499883883 conn(0x7f6e000a2bd0 msgr2=0x7f6e000a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- 172.21.15.45:0/2499883883 shutdown_connections 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- 172.21.15.45:0/2499883883 wait complete. 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: build_initial_monmap 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: monmap: 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:04:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 librados: starting msgr at 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 librados: starting objecter 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 Processor -- start 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- start start 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 librados: setting wanted keys 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 librados: calling monclient init 2024-09-16T03:04:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: init 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: _reopen_session rank -1 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): picked mon.noname-b con 0x7f6e0009fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): picked mon.noname-a con 0x7f6e00135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): picked mon.noname-c con 0x7f6e0013c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e00070680 con 0x7f6e00135950 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e0009d2c0 con 0x7f6e0009fc40 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e00071550 con 0x7f6e0013c010 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): _renew_subs 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e0863b640 10 monclient(hunting): authenticate will time out at 990.615620s 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e05bd6640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6e00135950 0x7f6e001cfe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38810/0 (socket says 172.21.15.45:38810) 2024-09-16T03:04:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e05bd6640 1 -- 172.21.15.45:0/1004476468 learned_addr learned my addr 172.21.15.45:0/1004476468 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6e06bd8640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6e0013c010 0x7f6e001d3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36166/0 (socket says 172.21.15.45:36166) 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6df2ffd640 10 client.?.objecter ms_handle_connect 0x7f6e0013c010 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6df2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6df2ffd640 10 client.?.objecter ms_handle_connect 0x7f6e00135950 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.962+0000 7f6df2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287675161 0 0) 0x7f6e00071550 con 0x7f6e0013c010 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): my global_id is 4262 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6de4004ec0 con 0x7f6e0013c010 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 client.?.objecter ms_handle_connect 0x7f6e0009fc40 2024-09-16T03:04:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4113484765 0 0) 0x7f6e00070680 con 0x7f6e00135950 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): my global_id is 4239 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e00071550 con 0x7f6e00135950 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3885156644 0 0) 0x7f6de4004ec0 con 0x7f6e0013c010 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e00070680 con 0x7f6e0013c010 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6df8001c40 con 0x7f6e0013c010 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): dump: 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.980 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1599413703 0 0) 0x7f6e00071550 con 0x7f6e00135950 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6de4004ec0 con 0x7f6e00135950 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065664257 0 0) 0x7f6e0009d2c0 con 0x7f6e0009fc40 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:04:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): my global_id is 4255 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e00071550 con 0x7f6e0009fc40 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6dfc001980 con 0x7f6e00135950 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient(hunting): dump: 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.982 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 948798584 0 0) 0x7f6e00070680 con 0x7f6e0013c010 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 >> v1:172.21.15.46:6789/0 conn(0x7f6e0009fc40 legacy=0x7f6e001cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 >> v1:172.21.15.45:6789/0 conn(0x7f6e00135950 legacy=0x7f6e001cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 monclient: found mon.c 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e00071830 con 0x7f6e0013c010 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:04:59.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 5 monclient: authenticate success, global_id 4262 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 10 monclient: _renew_subs 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0006fad0 con 0x7f6e0013c010 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 10 monclient: _renew_subs 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6e0006f3c0 con 0x7f6e0013c010 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 1 librados: init done 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6df80039a0 con 0x7f6e0013c010 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6df8003d70 con 0x7f6e0013c010 2024-09-16T03:04:59.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e04bd4640 4 set_mon_vals no callback set 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 monclient: dump: 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:04:59.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f6df8003f10 con 0x7f6e0013c010 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6de8005090 con 0x7f6e0013c010 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f6df804b1f0 con 0x7f6e0013c010 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 client.4262.objecter ms_dispatch 0x7f6e00000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 3 client.4262.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 3 client.4262.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 20 client.4262.objecter dump_active .. 0 homeless 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6e063d7640 10 monclient: get_auth_request con 0x7f6de403e290 auth_method 0 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.966+0000 7f6df2ffd640 10 client.4262.objecter ms_handle_connect 0x7f6de403e290 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.970+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f6df80165c0 con 0x7f6e0013c010 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.970+0000 7f6df2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:04:59.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:04:59.970+0000 7f6df2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.138+0000 7f6e0863b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:05:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.138+0000 7f6e0863b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:05:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.138+0000 7f6e0863b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.138+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6de8005380 con 0x7f6e0013c010 2024-09-16T03:05:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.138+0000 7f6df2ffd640 1 -- 172.21.15.45:0/1004476468 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10921 (unknown 941590788 0 1148292463) 0x7f6df801af90 con 0x7f6e0013c010 2024-09-16T03:05:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6df2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:05:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6df2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:00.145 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:05:00.145 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":19,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","created":"2024-09-16T03:04:38.421437+0000","modified":"2024-09-16T03:04:59.120592+0000","last_up_change":"2024-09-16T03:04:48.525305+0000","last_in_change":"2024-09-16T03:04:40.127320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:04:49.504411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:04:55.369938+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60a0a607-fe6c-4bd5-b369-e0dd6f6bd551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6800","nonce":3131549395}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6801","nonce":3131549395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6803","nonce":3131549395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6802","nonce":3131549395}]},"public_addr":"172.21.15.45:6800/3131549395","cluster_addr":"172.21.15.45:6801/3131549395","heartbeat_back_addr":"172.21.15.45:6803/3131549395","heartbeat_front_addr":"172.21.15.45:6802/3131549395","state":["exists","up"]},{"osd":1,"uuid":"10adca9c-3542-4901-a439-a1c9e9fcb8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6808","nonce":3228817795}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6809","nonce":3228817795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6811","nonce":3228817795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6810","nonce":3228817795}]},"public_addr":"172.21.15.45:6808/3228817795","cluster_addr":"172.21.15.45:6809/3228817795","heartbeat_back_addr":"172.21.15.45:6811/3228817795","heartbeat_front_addr":"172.21.15.45:6810/3228817795","state":["exists","up"]},{"osd":2,"uuid":"88fe1895-098a-4f13-95b2-4ead1739e71c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6804","nonce":207876738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6805","nonce":207876738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6807","nonce":207876738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6806","nonce":207876738}]},"public_addr":"172.21.15.45:6804/207876738","cluster_addr":"172.21.15.45:6805/207876738","heartbeat_back_addr":"172.21.15.45:6807/207876738","heartbeat_front_addr":"172.21.15.45:6806/207876738","state":["exists","up"]},{"osd":3,"uuid":"b04f94bc-5631-478e-904c-1dfc9a673a7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6808","nonce":3088806475}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6809","nonce":3088806475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6811","nonce":3088806475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6810","nonce":3088806475}]},"public_addr":"172.21.15.46:6808/3088806475","cluster_addr":"172.21.15.46:6809/3088806475","heartbeat_back_addr":"172.21.15.46:6811/3088806475","heartbeat_front_addr":"172.21.15.46:6810/3088806475","state":["exists","up"]},{"osd":4,"uuid":"c2dbca96-afb9-48ed-8a71-8a2bea37d152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6800","nonce":264199318}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":264199318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":264199318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":264199318}]},"public_addr":"172.21.15.46:6800/264199318","cluster_addr":"172.21.15.46:6801/264199318","heartbeat_back_addr":"172.21.15.46:6803/264199318","heartbeat_front_addr":"172.21.15.46:6802/264199318","state":["exists","up"]},{"osd":5,"uuid":"dabb8542-5498-4389-ad9e-0c0a54387723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":3390152579}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6805","nonce":3390152579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6807","nonce":3390152579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6806","nonce":3390152579}]},"public_addr":"172.21.15.46:6804/3390152579","cluster_addr":"172.21.15.46:6805/3390152579","heartbeat_back_addr":"172.21.15.46:6807/3390152579","heartbeat_front_addr":"172.21.15.46:6806/3390152579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.249472+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.780224+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.456157+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.830776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.656408+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.995384+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:05:00.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 10 librados: watch_flush enter 2024-09-16T03:05:00.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 10 librados: watch_flush exit 2024-09-16T03:05:00.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 20 client.4262.objecter shutdown clearing up homeless session... 2024-09-16T03:05:00.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 10 client.4262.objecter successfully canceled tick 2024-09-16T03:05:00.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 >> v1:172.21.15.46:6812/149360280 conn(0x7f6de403e290 legacy=0x7f6de4040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 10 monclient: shutdown 2024-09-16T03:05:00.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 >> v1:172.21.15.45:6790/0 conn(0x7f6e0013c010 legacy=0x7f6e001d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 shutdown_connections 2024-09-16T03:05:00.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 >> 172.21.15.45:0/1004476468 conn(0x7f6e000a2bd0 msgr2=0x7f6e0013edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 shutdown_connections 2024-09-16T03:05:00.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 1 -- 172.21.15.45:0/1004476468 wait complete. 2024-09-16T03:05:00.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.142+0000 7f6e0863b640 1 librados: shutdown 2024-09-16T03:05:00.161 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T03:05:00.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:05:00.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.266+0000 7f56378b6640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.266+0000 7f56378b6640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.266+0000 7f56378b6640 10 monclient: monmap: 2024-09-16T03:05:00.272 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.273 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.273 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.271708+0000 2024-09-16T03:05:00.273 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.271708+0000 2024-09-16T03:05:00.273 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.273 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.273 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.273 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.273 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.273 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 Processor -- start 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- start start 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: init 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): picked mon.noname-a con 0x7f563009b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): picked mon.noname-c con 0x7f563009b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): picked mon.noname-b con 0x7f563013b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f563009cc20 con 0x7f563009b8e0 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f563009d820 con 0x7f563009b250 2024-09-16T03:05:00.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f563009b6a0 con 0x7f563013b000 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): authenticate will time out at 990.920854s 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5635652640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f563009b8e0 0x7f563009bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38818/0 (socket says 172.21.15.45:38818) 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5635e53640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f563013b000 0x7f563013d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36178/0 (socket says 172.21.15.45:36178) 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5635652640 1 -- 172.21.15.45:0/299811101 learned_addr learned my addr 172.21.15.45:0/299811101 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706437903 0 0) 0x7f563009b6a0 con 0x7f563013b000 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient(hunting): my global_id is 4265 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f561c003530 con 0x7f563013b000 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923900073 0 0) 0x7f563009cc20 con 0x7f563009b8e0 2024-09-16T03:05:00.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient(hunting): my global_id is 4242 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f563009b6a0 con 0x7f563009b8e0 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 604785002 0 0) 0x7f561c003530 con 0x7f563013b000 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 >> v1:172.21.15.46:6789/0 conn(0x7f563009b250 legacy=0x7f56301398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 >> v1:172.21.15.45:6789/0 conn(0x7f563009b8e0 legacy=0x7f563009bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 monclient: found mon.noname-b 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56300a21f0 con 0x7f563013b000 2024-09-16T03:05:00.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5628001a20 con 0x7f563013b000 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: dump: 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.283 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 5 monclient: authenticate success, global_id 4265 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5628001bc0 con 0x7f563013b000 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 1 -- 172.21.15.45:0/299811101 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5628001fd0 con 0x7f563013b000 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 4 set_mon_vals no callback set 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5627fff640 10 monclient: dump: 2024-09-16T03:05:00.284 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: shutdown 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- 172.21.15.45:0/299811101 >> v1:172.21.15.45:6790/0 conn(0x7f563013b000 legacy=0x7f563013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- 172.21.15.45:0/299811101 shutdown_connections 2024-09-16T03:05:00.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- 172.21.15.45:0/299811101 >> 172.21.15.45:0/299811101 conn(0x7f56300a2bd0 msgr2=0x7f56300a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- 172.21.15.45:0/299811101 shutdown_connections 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- 172.21.15.45:0/299811101 wait complete. 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: monmap: 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.286 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 librados: starting msgr at 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 librados: starting objecter 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 Processor -- start 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- start start 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 librados: setting wanted keys 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 librados: calling monclient init 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: init 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): picked mon.noname-c con 0x7f563009b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): picked mon.noname-a con 0x7f563009b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): picked mon.noname-b con 0x7f563013b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56301316b0 con 0x7f563009b8e0 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5630131990 con 0x7f563013b000 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f563013a4b0 con 0x7f563009b250 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f56378b6640 10 monclient(hunting): authenticate will time out at 990.923857s 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5635652640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f563009b250 0x7f56301cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36192/0 (socket says 172.21.15.45:36192) 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5634e51640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f563009b8e0 0x7f56301d2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38832/0 (socket says 172.21.15.45:38832) 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.270+0000 7f5635652640 1 -- 172.21.15.45:0/2642190245 learned_addr learned my addr 172.21.15.45:0/2642190245 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 client.?.objecter ms_handle_connect 0x7f563009b8e0 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 client.?.objecter ms_handle_connect 0x7f563009b250 2024-09-16T03:05:00.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1986900149 0 0) 0x7f56301316b0 con 0x7f563009b8e0 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): my global_id is 4245 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5614004ef0 con 0x7f563009b8e0 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71025904 0 0) 0x7f563013a4b0 con 0x7f563009b250 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): my global_id is 4268 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56301316b0 con 0x7f563009b250 2024-09-16T03:05:00.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 client.?.objecter ms_handle_connect 0x7f563013b000 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3673283920 0 0) 0x7f5614004ef0 con 0x7f563009b8e0 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f563013a4b0 con 0x7f563009b8e0 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3904032915 0 0) 0x7f56301316b0 con 0x7f563009b250 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5614004ef0 con 0x7f563009b250 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5620001980 con 0x7f563009b8e0 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): dump: 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.290 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f562c0013c0 con 0x7f563009b250 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): dump: 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453427726 0 0) 0x7f5630131990 con 0x7f563013b000 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient(hunting): my global_id is 4261 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56301316b0 con 0x7f563013b000 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1012559201 0 0) 0x7f563013a4b0 con 0x7f563009b8e0 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 >> v1:172.21.15.45:6790/0 conn(0x7f563009b250 legacy=0x7f56301cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 >> v1:172.21.15.46:6789/0 conn(0x7f563013b000 legacy=0x7f56301d5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 monclient: found mon.a 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56300a5780 con 0x7f563009b8e0 2024-09-16T03:05:00.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 5 monclient: authenticate success, global_id 4245 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 10 monclient: _renew_subs 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f563009d2c0 con 0x7f563009b8e0 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 10 monclient: _renew_subs 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5630073aa0 con 0x7f563009b8e0 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 1 librados: init done 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5620001cc0 con 0x7f563009b8e0 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5620003a70 con 0x7f563009b8e0 2024-09-16T03:05:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56277fe640 4 set_mon_vals no callback set 2024-09-16T03:05:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 monclient: dump: 2024-09-16T03:05:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5610005090 con 0x7f563009b8e0 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f5620003c10 con 0x7f563009b8e0 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f562004b0b0 con 0x7f563009b8e0 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 client.4245.objecter ms_dispatch 0x7f5630000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 3 client.4245.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 3 client.4245.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 20 client.4245.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5635652640 10 monclient: get_auth_request con 0x7f561403e310 auth_method 0 2024-09-16T03:05:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.274+0000 7f5625ffb640 10 client.4245.objecter ms_handle_connect 0x7f561403e310 2024-09-16T03:05:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.278+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5620016ad0 con 0x7f563009b8e0 2024-09-16T03:05:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.278+0000 7f5625ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.278+0000 7f5625ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.442+0000 7f56378b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:05:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.442+0000 7f56378b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:05:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.442+0000 7f56378b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.442+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5610005380 con 0x7f563009b8e0 2024-09-16T03:05:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f5625ffb640 1 -- 172.21.15.45:0/2642190245 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10921 (unknown 941590788 0 1148292463) 0x7f56200163d0 con 0x7f563009b8e0 2024-09-16T03:05:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f5625ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:05:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f5625ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:05:00.449 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":19,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","created":"2024-09-16T03:04:38.421437+0000","modified":"2024-09-16T03:04:59.120592+0000","last_up_change":"2024-09-16T03:04:48.525305+0000","last_in_change":"2024-09-16T03:04:40.127320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:04:49.504411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:04:55.369938+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60a0a607-fe6c-4bd5-b369-e0dd6f6bd551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6800","nonce":3131549395}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6801","nonce":3131549395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6803","nonce":3131549395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6802","nonce":3131549395}]},"public_addr":"172.21.15.45:6800/3131549395","cluster_addr":"172.21.15.45:6801/3131549395","heartbeat_back_addr":"172.21.15.45:6803/3131549395","heartbeat_front_addr":"172.21.15.45:6802/3131549395","state":["exists","up"]},{"osd":1,"uuid":"10adca9c-3542-4901-a439-a1c9e9fcb8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6808","nonce":3228817795}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6809","nonce":3228817795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6811","nonce":3228817795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6810","nonce":3228817795}]},"public_addr":"172.21.15.45:6808/3228817795","cluster_addr":"172.21.15.45:6809/3228817795","heartbeat_back_addr":"172.21.15.45:6811/3228817795","heartbeat_front_addr":"172.21.15.45:6810/3228817795","state":["exists","up"]},{"osd":2,"uuid":"88fe1895-098a-4f13-95b2-4ead1739e71c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6804","nonce":207876738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6805","nonce":207876738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6807","nonce":207876738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6806","nonce":207876738}]},"public_addr":"172.21.15.45:6804/207876738","cluster_addr":"172.21.15.45:6805/207876738","heartbeat_back_addr":"172.21.15.45:6807/207876738","heartbeat_front_addr":"172.21.15.45:6806/207876738","state":["exists","up"]},{"osd":3,"uuid":"b04f94bc-5631-478e-904c-1dfc9a673a7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6808","nonce":3088806475}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6809","nonce":3088806475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6811","nonce":3088806475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6810","nonce":3088806475}]},"public_addr":"172.21.15.46:6808/3088806475","cluster_addr":"172.21.15.46:6809/3088806475","heartbeat_back_addr":"172.21.15.46:6811/3088806475","heartbeat_front_addr":"172.21.15.46:6810/3088806475","state":["exists","up"]},{"osd":4,"uuid":"c2dbca96-afb9-48ed-8a71-8a2bea37d152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6800","nonce":264199318}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":264199318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":264199318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":264199318}]},"public_addr":"172.21.15.46:6800/264199318","cluster_addr":"172.21.15.46:6801/264199318","heartbeat_back_addr":"172.21.15.46:6803/264199318","heartbeat_front_addr":"172.21.15.46:6802/264199318","state":["exists","up"]},{"osd":5,"uuid":"dabb8542-5498-4389-ad9e-0c0a54387723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":3390152579}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6805","nonce":3390152579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6807","nonce":3390152579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6806","nonce":3390152579}]},"public_addr":"172.21.15.46:6804/3390152579","cluster_addr":"172.21.15.46:6805/3390152579","heartbeat_back_addr":"172.21.15.46:6807/3390152579","heartbeat_front_addr":"172.21.15.46:6806/3390152579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.249472+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.780224+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.456157+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.830776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.656408+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.995384+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 10 librados: watch_flush enter 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 10 librados: watch_flush exit 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 20 client.4245.objecter shutdown clearing up homeless session... 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 10 client.4245.objecter successfully canceled tick 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 >> v1:172.21.15.46:6812/149360280 conn(0x7f561403e310 legacy=0x7f56140407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 10 monclient: shutdown 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 >> v1:172.21.15.45:6789/0 conn(0x7f563009b8e0 legacy=0x7f56301d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 shutdown_connections 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 >> 172.21.15.45:0/2642190245 conn(0x7f56300a2bd0 msgr2=0x7f5630139470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 shutdown_connections 2024-09-16T03:05:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 1 -- 172.21.15.45:0/2642190245 wait complete. 2024-09-16T03:05:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.446+0000 7f56378b6640 1 librados: shutdown 2024-09-16T03:05:00.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T03:05:00.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T03:05:00.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T03:05:00.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T03:05:00.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T03:05:00.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T03:05:00.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient: monmap: 2024-09-16T03:05:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.584275+0000 2024-09-16T03:05:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.584275+0000 2024-09-16T03:05:00.585 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.586 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.586 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.586 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.586 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.586 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient: monmap: 2024-09-16T03:05:00.586 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.584277+0000 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.584277+0000 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 1 Processor -- start 2024-09-16T03:05:00.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 1 Processor -- start 2024-09-16T03:05:00.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 1 -- start start 2024-09-16T03:05:00.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient: init 2024-09-16T03:05:00.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 1 -- start start 2024-09-16T03:05:00.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient: init 2024-09-16T03:05:00.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:00.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:00.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient(hunting): picked mon.noname-b con 0x7f99240a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient(hunting): picked mon.noname-c con 0x7f99240a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient(hunting): picked mon.noname-a con 0x7f9924073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f992409d9e0 con 0x7f9924073d40 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f992409dc20 con 0x7f99240a5810 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9924073360 con 0x7f99240a4d20 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient(hunting): authenticate will time out at 991.233507s 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient(hunting): picked mon.noname-b con 0x7fa210137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient(hunting): picked mon.noname-c con 0x7fa210138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient(hunting): picked mon.noname-a con 0x7fa21013e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2100a4f90 con 0x7fa21013e2a0 2024-09-16T03:05:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2100a4cb0 con 0x7fa210138100 2024-09-16T03:05:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20effd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fa210137d20 0x7fa2100a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:34962/0 (socket says 172.21.15.45:34962) 2024-09-16T03:05:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20effd640 1 -- 172.21.15.45:0/2221153651 learned_addr learned my addr 172.21.15.45:0/2221153651 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f9929eb5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9924073d40 0x7f9924140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42200/0 (socket says 172.21.15.45:42200) 2024-09-16T03:05:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f9929eb5640 1 -- 172.21.15.45:0/2878629422 learned_addr learned my addr 172.21.15.45:0/2878629422 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f99296b4640 1 --1- 172.21.15.45:0/2878629422 >> v1:172.21.15.45:6790/0 conn(0x7f99240a4d20 0x7f99240a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:34968/0 (socket says 172.21.15.45:34968) 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 1 -- 172.21.15.45:0/2221153651 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa21009dc70 con 0x7fa210137d20 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient(hunting): authenticate will time out at 991.233922s 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84601449 0 0) 0x7fa21009dc70 con 0x7fa210137d20 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): my global_id is 4271 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1f8003530 con 0x7fa210137d20 2024-09-16T03:05:00.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1521964805 0 0) 0x7f9924073360 con 0x7f99240a4d20 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): my global_id is 4274 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9910003530 con 0x7f99240a4d20 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162064092 0 0) 0x7f992409d9e0 con 0x7f9924073d40 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): my global_id is 4248 2024-09-16T03:05:00.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9924073360 con 0x7f9924073d40 2024-09-16T03:05:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1615923824 0 0) 0x7fa2100a4f90 con 0x7fa21013e2a0 2024-09-16T03:05:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): my global_id is 4251 2024-09-16T03:05:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa21009dc70 con 0x7fa21013e2a0 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4019448372 0 0) 0x7fa2100a4cb0 con 0x7fa210138100 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient(hunting): my global_id is 4264 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2100a4f90 con 0x7fa210138100 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 499437368 0 0) 0x7fa21009dc70 con 0x7fa21013e2a0 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 >> v1:172.21.15.45:6790/0 conn(0x7fa210137d20 legacy=0x7fa2100a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588472135 0 0) 0x7f992409dc20 con 0x7f99240a5810 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient(hunting): my global_id is 4267 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f992409d9e0 con 0x7f99240a5810 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 744245489 0 0) 0x7f9924073360 con 0x7f9924073d40 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 >> v1:172.21.15.45:6790/0 conn(0x7f99240a4d20 legacy=0x7f99240a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 >> v1:172.21.15.46:6789/0 conn(0x7f99240a5810 legacy=0x7f992413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 monclient: found mon.noname-a 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992409cf30 con 0x7f9924073d40 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f991c0018b0 con 0x7f9924073d40 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: dump: 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.598 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.598 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 5 monclient: authenticate success, global_id 4248 2024-09-16T03:05:00.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 >> v1:172.21.15.46:6789/0 conn(0x7fa210138100 legacy=0x7fa21013cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 monclient: found mon.noname-a 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21009cc50 con 0x7fa21013e2a0 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa208001a00 con 0x7fa21013e2a0 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: dump: 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.600 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.600 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.600 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.600 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.600 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.600 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.600 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 5 monclient: authenticate success, global_id 4251 2024-09-16T03:05:00.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f991c001a50 con 0x7f9924073d40 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 1 -- 172.21.15.45:0/2878629422 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f991c001ea0 con 0x7f9924073d40 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f991bfff640 10 monclient: dump: 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.602 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.602 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.602 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.602 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 4 set_mon_vals no callback set 2024-09-16T03:05:00.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa208001ba0 con 0x7fa21013e2a0 2024-09-16T03:05:00.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 1 -- 172.21.15.45:0/2221153651 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa208002030 con 0x7fa21013e2a0 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 4 set_mon_vals no callback set 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa20dffb640 10 monclient: dump: 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.603 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.604 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.604 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.604 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 10 monclient: shutdown 2024-09-16T03:05:00.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 1 -- 172.21.15.45:0/2221153651 >> v1:172.21.15.45:6789/0 conn(0x7fa21013e2a0 legacy=0x7fa210140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 10 monclient: shutdown 2024-09-16T03:05:00.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7f992b918640 1 -- 172.21.15.45:0/2878629422 >> v1:172.21.15.45:6789/0 conn(0x7f9924073d40 legacy=0x7f9924140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 1 -- 172.21.15.45:0/2221153651 shutdown_connections 2024-09-16T03:05:00.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.582+0000 7fa21570d640 1 -- 172.21.15.45:0/2221153651 >> 172.21.15.45:0/2221153651 conn(0x7fa2100aa110 msgr2=0x7fa2100aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 -- 172.21.15.45:0/2878629422 shutdown_connections 2024-09-16T03:05:00.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 -- 172.21.15.45:0/2878629422 >> 172.21.15.45:0/2878629422 conn(0x7f99240aa720 msgr2=0x7f99240aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 -- 172.21.15.45:0/2878629422 shutdown_connections 2024-09-16T03:05:00.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 -- 172.21.15.45:0/2878629422 wait complete. 2024-09-16T03:05:00.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 -- 172.21.15.45:0/2221153651 shutdown_connections 2024-09-16T03:05:00.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 -- 172.21.15.45:0/2221153651 wait complete. 2024-09-16T03:05:00.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient: monmap: 2024-09-16T03:05:00.610 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.610 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 librados: starting msgr at 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 librados: starting objecter 2024-09-16T03:05:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 Processor -- start 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient: monmap: 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.612 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.613 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 librados: starting msgr at 2024-09-16T03:05:00.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 librados: starting objecter 2024-09-16T03:05:00.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 Processor -- start 2024-09-16T03:05:00.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 -- start start 2024-09-16T03:05:00.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 librados: setting wanted keys 2024-09-16T03:05:00.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 librados: calling monclient init 2024-09-16T03:05:00.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient: init 2024-09-16T03:05:00.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 -- start start 2024-09-16T03:05:00.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 librados: setting wanted keys 2024-09-16T03:05:00.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 librados: calling monclient init 2024-09-16T03:05:00.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient: init 2024-09-16T03:05:00.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:00.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient(hunting): picked mon.noname-c con 0x7f9924073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient(hunting): picked mon.noname-b con 0x7f99240a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient(hunting): picked mon.noname-a con 0x7f99240a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99240a0580 con 0x7f99240a5810 2024-09-16T03:05:00.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99240a1ce0 con 0x7f99240a4d20 2024-09-16T03:05:00.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99240a17b0 con 0x7f9924073d40 2024-09-16T03:05:00.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f992b918640 10 monclient(hunting): authenticate will time out at 991.237773s 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient(hunting): picked mon.noname-a con 0x7fa210137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient(hunting): picked mon.noname-b con 0x7fa210138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient(hunting): picked mon.noname-c con 0x7fa21013e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2100a42f0 con 0x7fa210137d20 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa21013f4a0 con 0x7fa210138100 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa210141af0 con 0x7fa21013e2a0 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient(hunting): authenticate will time out at 991.237948s 2024-09-16T03:05:00.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f99296b4640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9924073d40 0x7f9924134140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:34982/0 (socket says 172.21.15.45:34982) 2024-09-16T03:05:00.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f99296b4640 1 -- 172.21.15.45:0/766179941 learned_addr learned my addr 172.21.15.45:0/766179941 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9928eb3640 1 --1- 172.21.15.45:0/766179941 >> v1:172.21.15.46:6789/0 conn(0x7f99240a4d20 0x7f99241cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.45:39490/0 (socket says 172.21.15.45:39490) 2024-09-16T03:05:00.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa20effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa210137d20 0x7fa21013e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42208/0 (socket says 172.21.15.45:42208) 2024-09-16T03:05:00.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa20effd640 1 -- 172.21.15.45:0/1138284184 learned_addr learned my addr 172.21.15.45:0/1138284184 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 client.?.objecter ms_handle_connect 0x7f99240a4d20 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 client.?.objecter ms_handle_connect 0x7f9924073d40 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 client.?.objecter ms_handle_connect 0x7fa21013e2a0 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 client.?.objecter ms_handle_connect 0x7fa210138100 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3139508809 0 0) 0x7fa210141af0 con 0x7fa21013e2a0 2024-09-16T03:05:00.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): my global_id is 4280 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1fc004ec0 con 0x7fa21013e2a0 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 client.?.objecter ms_handle_connect 0x7fa210137d20 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 client.?.objecter ms_handle_connect 0x7f99240a5810 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1495430206 0 0) 0x7f99240a0580 con 0x7f99240a5810 2024-09-16T03:05:00.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): my global_id is 4254 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9908004ec0 con 0x7f99240a5810 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1845523250 0 0) 0x7f9914001300 con 0x7f99240a4d20 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): my global_id is 4270 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99240a0580 con 0x7f99240a4d20 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376500616 0 0) 0x7f9910003530 con 0x7f9924073d40 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): my global_id is 4277 2024-09-16T03:05:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9914001300 con 0x7f9924073d40 2024-09-16T03:05:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516588613 0 0) 0x7fa21013f4a0 con 0x7fa210138100 2024-09-16T03:05:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): my global_id is 4273 2024-09-16T03:05:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa210141af0 con 0x7fa210138100 2024-09-16T03:05:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1277647036 0 0) 0x7f9908004ec0 con 0x7f99240a5810 2024-09-16T03:05:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2477506583 0 0) 0x7fa1fc004ec0 con 0x7fa21013e2a0 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa21013f4a0 con 0x7fa21013e2a0 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa208001ef0 con 0x7fa21013e2a0 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): dump: 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.624 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1242892939 0 0) 0x7fa2100a42f0 con 0x7fa210137d20 2024-09-16T03:05:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient(hunting): my global_id is 4257 2024-09-16T03:05:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1fc004ec0 con 0x7fa210137d20 2024-09-16T03:05:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9910003530 con 0x7f99240a5810 2024-09-16T03:05:00.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3051686635 0 0) 0x7fa21013f4a0 con 0x7fa21013e2a0 2024-09-16T03:05:00.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 >> v1:172.21.15.46:6789/0 conn(0x7fa210138100 legacy=0x7fa21013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f991c001da0 con 0x7f99240a5810 2024-09-16T03:05:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 >> v1:172.21.15.45:6789/0 conn(0x7fa210137d20 legacy=0x7fa21013e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 monclient: found mon.c 2024-09-16T03:05:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2100a1300 con 0x7fa21013e2a0 2024-09-16T03:05:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa1f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): dump: 2024-09-16T03:05:00.627 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.627 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.627 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.627 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.627 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.628 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.628 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.628 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.628 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.628 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 5 monclient: authenticate success, global_id 4280 2024-09-16T03:05:00.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient: _renew_subs 2024-09-16T03:05:00.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa21013ff30 con 0x7fa21013e2a0 2024-09-16T03:05:00.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient: _renew_subs 2024-09-16T03:05:00.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa210140ad0 con 0x7fa21013e2a0 2024-09-16T03:05:00.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2967739737 0 0) 0x7f9914001300 con 0x7f9924073d40 2024-09-16T03:05:00.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7fa21570d640 1 librados: init done 2024-09-16T03:05:00.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9908004ec0 con 0x7f9924073d40 2024-09-16T03:05:00.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 994891019 0 0) 0x7f99240a0580 con 0x7f99240a4d20 2024-09-16T03:05:00.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9914001300 con 0x7f99240a4d20 2024-09-16T03:05:00.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f99200018e0 con 0x7f9924073d40 2024-09-16T03:05:00.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): dump: 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.632 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9914001bd0 con 0x7f99240a4d20 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.586+0000 7f9919ffb640 10 monclient(hunting): dump: 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.633 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2929922319 0 0) 0x7f9910003530 con 0x7f99240a5810 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 >> v1:172.21.15.45:6790/0 conn(0x7f9924073d40 legacy=0x7f9924134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 >> v1:172.21.15.46:6789/0 conn(0x7f99240a4d20 legacy=0x7f99241cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 monclient: found mon.a 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9924120b10 con 0x7f99240a5810 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa21570d640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa21570d640 15 client.4280.objecter _session_command_op_assign -1 1 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa21570d640 20 client.4280.objecter _get_session osd=-1 returning homeless 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa21570d640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa21570d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.45:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fa2100770b0 con 0x7fa21013e2a0 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa208001b00 con 0x7fa21013e2a0 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa208003e70 con 0x7fa21013e2a0 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 monclient: dump: 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fa20800fee0 con 0x7fa21013e2a0 2024-09-16T03:05:00.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa20d7fa640 4 set_mon_vals no callback set 2024-09-16T03:05:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 5 monclient: authenticate success, global_id 4254 2024-09-16T03:05:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 10 monclient: _renew_subs 2024-09-16T03:05:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f991c004e20 con 0x7f99240a5810 2024-09-16T03:05:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f992409f850 con 0x7f99240a5810 2024-09-16T03:05:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 10 monclient: _renew_subs 2024-09-16T03:05:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f99241c5c40 con 0x7f99240a5810 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 1 librados: init done 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 10 client.4254.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 15 client.4254.objecter _session_command_op_assign -1 1 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 20 client.4254.objecter _get_session osd=-1 returning homeless 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.45:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f992409d6d0 con 0x7f99240a5810 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fa20804a2f0 con 0x7fa21013e2a0 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 client.4280.objecter ms_dispatch 0x7fa210000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 3 client.4280.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 3 client.4280.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f991c005170 con 0x7f99240a5810 2024-09-16T03:05:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f991b7fe640 4 set_mon_vals no callback set 2024-09-16T03:05:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 monclient: dump: 2024-09-16T03:05:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f991c0111e0 con 0x7f99240a5810 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 client.4280.objecter checking command 1 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 20 client.4280.objecter _get_session s=0x7fa1fc044840 osd=0 3 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 20 client.4280.objecter put_session s=0x7fa1fc044840 osd=0 3 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 15 client.4280.objecter _session_command_op_remove -1 1 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 20 client.4280.objecter _get_session s=0x7fa1fc044840 osd=0 3 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 20 client.4280.objecter get_session s=0x7fa1fc044840 osd=0 3 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 15 client.4280.objecter _session_command_op_assign 0 1 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 20 client.4280.objecter put_session s=0x7fa1fc044840 osd=0 4 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 client.4280.objecter _send_command 1 2024-09-16T03:05:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.45:6800/3131549395 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa1fc048ba0 con 0x7fa1fc044ef0 2024-09-16T03:05:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 20 client.4280.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fa208011bb0 con 0x7fa21013e2a0 2024-09-16T03:05:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T03:05:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa20e7fc640 10 monclient: get_auth_request con 0x7fa1fc044ef0 auth_method 0 2024-09-16T03:05:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa20effd640 10 monclient: get_auth_request con 0x7fa1fc03e360 auth_method 0 2024-09-16T03:05:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 client.4280.objecter ms_handle_connect 0x7fa1fc044ef0 2024-09-16T03:05:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 client.4280.objecter ms_handle_connect 0x7fa1fc03e360 2024-09-16T03:05:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f991c04b1f0 con 0x7f99240a5810 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 client.4254.objecter ms_dispatch 0x7f9924000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 3 client.4254.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 3 client.4254.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 client.4254.objecter checking command 1 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f99296b4640 10 monclient: get_auth_request con 0x7f990803e550 auth_method 0 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 20 client.4254.objecter _get_session s=0x7f9908044b00 osd=5 3 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 20 client.4254.objecter put_session s=0x7f9908044b00 osd=5 3 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 15 client.4254.objecter _session_command_op_remove -1 1 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 20 client.4254.objecter _get_session s=0x7f9908044b00 osd=5 3 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 20 client.4254.objecter get_session s=0x7f9908044b00 osd=5 3 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 15 client.4254.objecter _session_command_op_assign 5 1 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 20 client.4254.objecter put_session s=0x7f9908044b00 osd=5 4 2024-09-16T03:05:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 client.4254.objecter _send_command 1 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.46:6804/3390152579 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f991c001a50 con 0x7f99080451b0 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9928eb3640 10 monclient: get_auth_request con 0x7f99080451b0 auth_method 0 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 20 client.4254.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 client.4254.objecter ms_handle_connect 0x7f990803e550 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f991c04b5a0 con 0x7f99240a5810 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== osd.0 v1:172.21.15.45:6800/3131549395 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa1fc048ba0 con 0x7fa1fc044ef0 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 client.4280.objecter ms_dispatch 0x7fa210000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 10 client.4280.objecter _finish_command 1 = system:0 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 20 client.4280.objecter put_session s=0x7fa1fc044840 osd=0 4 2024-09-16T03:05:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7fa1f77fe640 15 client.4280.objecter _session_command_op_remove 0 1 2024-09-16T03:05:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.590+0000 7f9919ffb640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T03:05:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.598+0000 7f9919ffb640 10 client.4254.objecter ms_handle_connect 0x7f99080451b0 2024-09-16T03:05:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.598+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== osd.5 v1:172.21.15.46:6804/3390152579 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f991c001a50 con 0x7f99080451b0 2024-09-16T03:05:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.598+0000 7f9919ffb640 10 client.4254.objecter ms_dispatch 0x7f9924000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:05:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.598+0000 7f9919ffb640 10 client.4254.objecter _finish_command 1 = system:0 2024-09-16T03:05:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.598+0000 7f9919ffb640 20 client.4254.objecter put_session s=0x7f9908044b00 osd=5 4 2024-09-16T03:05:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.598+0000 7f9919ffb640 15 client.4254.objecter _session_command_op_remove 5 1 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 15 client.4280.objecter _session_command_op_assign -1 2 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 20 client.4280.objecter _get_session s=0x7fa1fc044840 osd=0 3 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 20 client.4280.objecter put_session s=0x7fa1fc044840 osd=0 3 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 20 client.4280.objecter _get_session s=0x7fa1fc044840 osd=0 3 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 15 client.4280.objecter _session_command_op_remove -1 2 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 20 client.4280.objecter get_session s=0x7fa1fc044840 osd=0 3 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 15 client.4280.objecter _session_command_op_assign 0 2 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 20 client.4280.objecter put_session s=0x7fa1fc044840 osd=0 4 2024-09-16T03:05:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 10 client.4280.objecter _send_command 2 2024-09-16T03:05:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 --> v1:172.21.15.45:6800/3131549395 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa21009efb0 con 0x7fa1fc044ef0 2024-09-16T03:05:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa1f77fe640 1 -- 172.21.15.45:0/1138284184 <== osd.0 v1:172.21.15.45:6800/3131549395 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3580782990) 0x7fa21009efb0 con 0x7fa1fc044ef0 2024-09-16T03:05:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa1f77fe640 10 client.4280.objecter ms_dispatch 0x7fa210000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:05:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa1f77fe640 10 client.4280.objecter _finish_command 2 = system:0 2024-09-16T03:05:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa1f77fe640 20 client.4280.objecter put_session s=0x7fa1fc044840 osd=0 4 2024-09-16T03:05:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa1f77fe640 15 client.4280.objecter _session_command_op_remove 0 2 2024-09-16T03:05:00.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 10 librados: watch_flush enter 2024-09-16T03:05:00.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 10 librados: watch_flush exit 2024-09-16T03:05:00.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 10 client.4280.objecter close_session for osd.0 2024-09-16T03:05:00.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 >> v1:172.21.15.45:6800/3131549395 conn(0x7fa1fc044ef0 legacy=0x7fa1fc047330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 20 client.4280.objecter put_session s=0x7fa1fc044840 osd=0 1 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 20 client.4280.objecter shutdown clearing up homeless session... 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 10 client.4280.objecter successfully canceled tick 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 >> v1:172.21.15.46:6812/149360280 conn(0x7fa1fc03e360 legacy=0x7fa1fc040820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 10 monclient: shutdown 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 >> v1:172.21.15.45:6790/0 conn(0x7fa21013e2a0 legacy=0x7fa2101d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa20f7fe640 1 -- 172.21.15.45:0/1138284184 reap_dead start 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 shutdown_connections 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 >> 172.21.15.45:0/1138284184 conn(0x7fa2100aa110 msgr2=0x7fa2100731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 shutdown_connections 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 1 -- 172.21.15.45:0/1138284184 wait complete. 2024-09-16T03:05:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.606+0000 7fa21570d640 1 librados: shutdown 2024-09-16T03:05:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 10 client.4254.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:05:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 15 client.4254.objecter _session_command_op_assign -1 2 2024-09-16T03:05:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 20 client.4254.objecter _get_session s=0x7f9908044b00 osd=5 3 2024-09-16T03:05:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 20 client.4254.objecter put_session s=0x7f9908044b00 osd=5 3 2024-09-16T03:05:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 20 client.4254.objecter _get_session s=0x7f9908044b00 osd=5 3 2024-09-16T03:05:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 15 client.4254.objecter _session_command_op_remove -1 2 2024-09-16T03:05:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 20 client.4254.objecter get_session s=0x7f9908044b00 osd=5 3 2024-09-16T03:05:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 15 client.4254.objecter _session_command_op_assign 5 2 2024-09-16T03:05:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 20 client.4254.objecter put_session s=0x7f9908044b00 osd=5 4 2024-09-16T03:05:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 10 client.4254.objecter _send_command 2 2024-09-16T03:05:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 --> v1:172.21.15.46:6804/3390152579 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f992406edb0 con 0x7f99080451b0 2024-09-16T03:05:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f9919ffb640 1 -- 172.21.15.45:0/766179941 <== osd.5 v1:172.21.15.46:6804/3390152579 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7f992406edb0 con 0x7f99080451b0 2024-09-16T03:05:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f9919ffb640 10 client.4254.objecter ms_dispatch 0x7f9924000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:05:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f9919ffb640 10 client.4254.objecter _finish_command 2 = system:0 2024-09-16T03:05:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f9919ffb640 20 client.4254.objecter put_session s=0x7f9908044b00 osd=5 4 2024-09-16T03:05:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.610+0000 7f9919ffb640 15 client.4254.objecter _session_command_op_remove 5 2 2024-09-16T03:05:00.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 10 librados: watch_flush enter 2024-09-16T03:05:00.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 10 librados: watch_flush exit 2024-09-16T03:05:00.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 10 client.4254.objecter close_session for osd.5 2024-09-16T03:05:00.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 >> v1:172.21.15.46:6804/3390152579 conn(0x7f99080451b0 legacy=0x7f99080475f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 20 client.4254.objecter put_session s=0x7f9908044b00 osd=5 1 2024-09-16T03:05:00.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 20 client.4254.objecter shutdown clearing up homeless session... 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 10 client.4254.objecter successfully canceled tick 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 >> v1:172.21.15.46:6812/149360280 conn(0x7f990803e550 legacy=0x7f9908040a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 10 monclient: shutdown 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 >> v1:172.21.15.45:6789/0 conn(0x7f99240a5810 legacy=0x7f99241d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f9929eb5640 1 -- 172.21.15.45:0/766179941 reap_dead start 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 shutdown_connections 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 >> 172.21.15.45:0/766179941 conn(0x7f99240aa720 msgr2=0x7f99241381b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 shutdown_connections 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 1 -- 172.21.15.45:0/766179941 wait complete. 2024-09-16T03:05:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.614+0000 7f992b918640 1 librados: shutdown 2024-09-16T03:05:00.655 INFO:teuthology.orchestra.run.smithi045.stdout:47244640261 2024-09-16T03:05:00.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:05:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.630+0000 7fdd3a349640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.630+0000 7fdd3a349640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.630+0000 7fdd3a349640 10 monclient: monmap: 2024-09-16T03:05:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.635747+0000 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.635747+0000 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 Processor -- start 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 -- start start 2024-09-16T03:05:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient: init 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient: monmap: 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.636663+0000 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.636663+0000 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.659 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient(hunting): picked mon.noname-a con 0x7fdd340a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient(hunting): picked mon.noname-c con 0x7fdd340a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient(hunting): picked mon.noname-b con 0x7fdd34073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3409d9e0 con 0x7fdd340a4d20 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3409dc20 con 0x7fdd340a5810 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd34073360 con 0x7fdd34073d40 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient(hunting): authenticate will time out at 991.284806s 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd388e6640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdd34073d40 0x7fdd34140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35010/0 (socket says 172.21.15.45:35010) 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd388e6640 1 -- 172.21.15.45:0/1410455372 learned_addr learned my addr 172.21.15.45:0/1410455372 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3422111813 0 0) 0x7fdd34073360 con 0x7fdd34073d40 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): my global_id is 4283 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd1c003530 con 0x7fdd34073d40 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3776480412 0 0) 0x7fdd3409d9e0 con 0x7fdd340a4d20 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): my global_id is 4260 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd34073360 con 0x7fdd340a4d20 2024-09-16T03:05:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 1 Processor -- start 2024-09-16T03:05:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503601926 0 0) 0x7fdd3409dc20 con 0x7fdd340a5810 2024-09-16T03:05:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient(hunting): my global_id is 4276 2024-09-16T03:05:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd3409d9e0 con 0x7fdd340a5810 2024-09-16T03:05:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3947129798 0 0) 0x7fdd1c003530 con 0x7fdd34073d40 2024-09-16T03:05:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 >> v1:172.21.15.46:6789/0 conn(0x7fdd340a5810 legacy=0x7fdd3413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 >> v1:172.21.15.45:6789/0 conn(0x7fdd340a4d20 legacy=0x7fdd340a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 1 -- start start 2024-09-16T03:05:00.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient: init 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 monclient: found mon.noname-b 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3409cf30 con 0x7fdd34073d40 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdd2c001a20 con 0x7fdd34073d40 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 5 monclient: authenticate success, global_id 4283 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: dump: 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient: monmap: 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.638109+0000 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.638109+0000 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.667 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd2c001c40 con 0x7fdd34073d40 2024-09-16T03:05:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 1 -- 172.21.15.45:0/1410455372 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdd2c002090 con 0x7fdd34073d40 2024-09-16T03:05:00.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 4 set_mon_vals no callback set 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd32ffd640 10 monclient: dump: 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.668 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient(hunting): picked mon.noname-b con 0x7f731c0a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 1 Processor -- start 2024-09-16T03:05:00.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient(hunting): picked mon.noname-c con 0x7f731c0a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 1 -- start start 2024-09-16T03:05:00.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient: init 2024-09-16T03:05:00.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient: shutdown 2024-09-16T03:05:00.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient(hunting): picked mon.noname-a con 0x7f731c073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 -- 172.21.15.45:0/1410455372 >> v1:172.21.15.45:6790/0 conn(0x7fdd34073d40 legacy=0x7fdd34140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 -- 172.21.15.45:0/1410455372 shutdown_connections 2024-09-16T03:05:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 -- 172.21.15.45:0/1410455372 >> 172.21.15.45:0/1410455372 conn(0x7fdd340aa720 msgr2=0x7fdd340aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731ad9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f731c073d40 0x7f731c140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42236/0 (socket says 172.21.15.45:42236) 2024-09-16T03:05:00.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731ad9d640 1 -- 172.21.15.45:0/4229931020 learned_addr learned my addr 172.21.15.45:0/4229931020 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.672 INFO:teuthology.orchestra.run.smithi045.stdout:51539607556 2024-09-16T03:05:00.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:05:00.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 -- 172.21.15.45:0/1410455372 shutdown_connections 2024-09-16T03:05:00.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 1 -- 172.21.15.45:0/4229931020 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f731c09d9e0 con 0x7f731c073d40 2024-09-16T03:05:00.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 1 -- 172.21.15.45:0/4229931020 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f731c09dc20 con 0x7f731c0a5810 2024-09-16T03:05:00.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 1 -- 172.21.15.45:0/4229931020 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f731c073360 con 0x7f731c0a4d20 2024-09-16T03:05:00.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 10 monclient(hunting): authenticate will time out at 991.287040s 2024-09-16T03:05:00.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 1 -- 172.21.15.45:0/1410455372 wait complete. 2024-09-16T03:05:00.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993886483 0 0) 0x7f731c09d9e0 con 0x7f731c073d40 2024-09-16T03:05:00.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient(hunting): my global_id is 4263 2024-09-16T03:05:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7304003530 con 0x7f731c073d40 2024-09-16T03:05:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:00.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fce2c0a3840 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient(hunting): picked mon.noname-b con 0x7fce2c09daa0 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient(hunting): picked mon.noname-a con 0x7fce2c09d250 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce2c069630 con 0x7fce2c09d250 2024-09-16T03:05:00.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce2c0a4330 con 0x7fce2c0a3840 2024-09-16T03:05:00.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce2c0a4500 con 0x7fce2c09daa0 2024-09-16T03:05:00.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce32bfd640 10 monclient(hunting): authenticate will time out at 991.287569s 2024-09-16T03:05:00.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815497046 0 0) 0x7f731c073360 con 0x7f731c0a4d20 2024-09-16T03:05:00.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient(hunting): my global_id is 4286 2024-09-16T03:05:00.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f731c09d9e0 con 0x7f731c0a4d20 2024-09-16T03:05:00.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce313fa640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fce2c09daa0 0x7fce2c13aab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35030/0 (socket says 172.21.15.45:35030) 2024-09-16T03:05:00.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fce313fa640 1 -- 172.21.15.45:0/4151874292 learned_addr learned my addr 172.21.15.45:0/4151874292 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3178347408 0 0) 0x7f7304003530 con 0x7f731c073d40 2024-09-16T03:05:00.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 >> v1:172.21.15.45:6790/0 conn(0x7f731c0a4d20 legacy=0x7f731c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 >> v1:172.21.15.46:6789/0 conn(0x7f731c0a5810 legacy=0x7f731c13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 monclient: found mon.noname-a 2024-09-16T03:05:00.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f731c09cf30 con 0x7f731c073d40 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f730c0016a0 con 0x7f731c073d40 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f731959a640 10 monclient: dump: 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.679 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7fdd3a349640 10 monclient: monmap: 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.680 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.681 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.681 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.681 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.681 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.681 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 5 monclient: authenticate success, global_id 4263 2024-09-16T03:05:00.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.634+0000 7f7320b75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1813549114 0 0) 0x7fce2c0a4500 con 0x7fce2c09daa0 2024-09-16T03:05:00.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): my global_id is 4289 2024-09-16T03:05:00.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce18003530 con 0x7fce2c09daa0 2024-09-16T03:05:00.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 librados: starting msgr at 2024-09-16T03:05:00.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 librados: starting objecter 2024-09-16T03:05:00.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 Processor -- start 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129313848 0 0) 0x7fce2c069630 con 0x7fce2c09d250 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): my global_id is 4266 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce2c0a4500 con 0x7fce2c09d250 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936309167 0 0) 0x7fce2c0a4330 con 0x7fce2c0a3840 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient(hunting): my global_id is 4282 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce2c069630 con 0x7fce2c0a3840 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f730c001840 con 0x7f731c073d40 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f731959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f731959a640 1 -- 172.21.15.45:0/4229931020 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f730c001cd0 con 0x7f731c073d40 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f731959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f731959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f731959a640 10 monclient: dump: 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.684 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.685 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.685 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.685 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.685 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.685 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.685 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.685 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2213133119 0 0) 0x7fce18003530 con 0x7fce2c09daa0 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 >> v1:172.21.15.46:6789/0 conn(0x7fce2c0a3840 legacy=0x7fce2c0a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 >> v1:172.21.15.45:6789/0 conn(0x7fce2c09d250 legacy=0x7fce2c13e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 monclient: found mon.noname-b 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2c142930 con 0x7fce2c09daa0 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 5 monclient: authenticate success, global_id 4289 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fce1c0016a0 con 0x7fce2c09daa0 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: dump: 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.689 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce1c001900 con 0x7fce2c09daa0 2024-09-16T03:05:00.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 4 set_mon_vals no callback set 2024-09-16T03:05:00.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 -- start start 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 librados: setting wanted keys 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 librados: calling monclient init 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient: init 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient(hunting): picked mon.noname-a con 0x7fdd34073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient(hunting): picked mon.noname-c con 0x7fdd340a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient(hunting): picked mon.noname-b con 0x7fdd340a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd34140c20 con 0x7fdd34073d40 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd34140470 con 0x7fdd340a5810 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3413d7a0 con 0x7fdd340a4d20 2024-09-16T03:05:00.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient(hunting): authenticate will time out at 991.288930s 2024-09-16T03:05:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 1 -- 172.21.15.45:0/4151874292 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fce1c001c50 con 0x7fce2c09daa0 2024-09-16T03:05:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce30bf9640 10 monclient: dump: 2024-09-16T03:05:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.692 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 4 set_mon_vals no callback set 2024-09-16T03:05:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient: shutdown 2024-09-16T03:05:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 -- 172.21.15.45:0/4229931020 >> v1:172.21.15.45:6789/0 conn(0x7f731c073d40 legacy=0x7f731c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd388e6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdd340a5810 0x7fdd341d51d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.45:39536/0 (socket says 172.21.15.45:39536) 2024-09-16T03:05:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd388e6640 1 -- 172.21.15.45:0/2771466812 learned_addr learned my addr 172.21.15.45:0/2771466812 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 -- 172.21.15.45:0/4229931020 shutdown_connections 2024-09-16T03:05:00.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 -- 172.21.15.45:0/4229931020 >> 172.21.15.45:0/4229931020 conn(0x7f731c0aa720 msgr2=0x7f731c0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 -- 172.21.15.45:0/4229931020 shutdown_connections 2024-09-16T03:05:00.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 client.?.objecter ms_handle_connect 0x7fdd340a5810 2024-09-16T03:05:00.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient: shutdown 2024-09-16T03:05:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 -- 172.21.15.45:0/4151874292 >> v1:172.21.15.45:6790/0 conn(0x7fce2c09daa0 legacy=0x7fce2c13aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 -- 172.21.15.45:0/4151874292 shutdown_connections 2024-09-16T03:05:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 -- 172.21.15.45:0/4151874292 >> 172.21.15.45:0/4151874292 conn(0x7fce2c0a5310 msgr2=0x7fce2c068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 756501654 0 0) 0x7fdd34140470 con 0x7fdd340a5810 2024-09-16T03:05:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): my global_id is 4285 2024-09-16T03:05:00.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd20004ec0 con 0x7fdd340a5810 2024-09-16T03:05:00.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 client.?.objecter ms_handle_connect 0x7fdd340a4d20 2024-09-16T03:05:00.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 -- 172.21.15.45:0/4151874292 shutdown_connections 2024-09-16T03:05:00.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 client.?.objecter ms_handle_connect 0x7fdd34073d40 2024-09-16T03:05:00.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 -- 172.21.15.45:0/4151874292 wait complete. 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1830454075 0 0) 0x7fdd20004ec0 con 0x7fdd340a5810 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd34140470 con 0x7fdd340a5810 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdd2c001f90 con 0x7fdd340a5810 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): dump: 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.697 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2499776692 0 0) 0x7fdd3413d7a0 con 0x7fdd340a4d20 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): my global_id is 4292 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd20004ec0 con 0x7fdd340a4d20 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3520122255 0 0) 0x7fdd34073360 con 0x7fdd34073d40 2024-09-16T03:05:00.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient(hunting): my global_id is 4269 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd3413d7a0 con 0x7fdd34073d40 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient: monmap: 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.699 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.700 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2414914563 0 0) 0x7fdd34140470 con 0x7fdd340a5810 2024-09-16T03:05:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 >> v1:172.21.15.45:6790/0 conn(0x7fdd340a4d20 legacy=0x7fdd3413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 >> v1:172.21.15.45:6789/0 conn(0x7fdd34073d40 legacy=0x7fdd3413c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 monclient: found mon.b 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd34140750 con 0x7fdd340a5810 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 5 monclient: authenticate success, global_id 4285 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient: _renew_subs 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd340a23b0 con 0x7fdd340a5810 2024-09-16T03:05:00.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient: _renew_subs 2024-09-16T03:05:00.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:00.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd34140190 con 0x7fdd340a5810 2024-09-16T03:05:00.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 librados: init done 2024-09-16T03:05:00.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 -- 172.21.15.45:0/4229931020 wait complete. 2024-09-16T03:05:00.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 librados: starting msgr at 2024-09-16T03:05:00.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 librados: starting objecter 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient: monmap: 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.703 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 librados: starting msgr at 2024-09-16T03:05:00.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 librados: starting objecter 2024-09-16T03:05:00.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 Processor -- start 2024-09-16T03:05:00.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 Processor -- start 2024-09-16T03:05:00.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 -- start start 2024-09-16T03:05:00.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 librados: setting wanted keys 2024-09-16T03:05:00.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 librados: calling monclient init 2024-09-16T03:05:00.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient: init 2024-09-16T03:05:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.1 v1:172.21.15.46:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd2c001590 con 0x7fdd340a5810 2024-09-16T03:05:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdd2c0051a0 con 0x7fdd340a5810 2024-09-16T03:05:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient(hunting): picked mon.noname-b con 0x7fce2c09d250 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient(hunting): picked mon.noname-a con 0x7fce2c0a3840 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fce2c1d2ab0 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce2c073d10 con 0x7fce2c0a3840 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce2c072e10 con 0x7fce2c09d250 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd30ff9640 10 monclient: dump: 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd327fc640 4 set_mon_vals no callback set 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 client.4285.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 15 client.4285.objecter _session_command_op_assign -1 1 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 20 client.4285.objecter _get_session osd=-1 returning homeless 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.46:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fdd14002920 con 0x7fdd340a5810 2024-09-16T03:05:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce2c0a79b0 con 0x7fce2c1d2ab0 2024-09-16T03:05:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce32bfd640 10 monclient(hunting): authenticate will time out at 991.291508s 2024-09-16T03:05:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce31bfb640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fce2c09d250 0x7fce2c070230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.45:39550/0 (socket says 172.21.15.45:39550) 2024-09-16T03:05:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7fce31bfb640 1 -- 172.21.15.45:0/1259077251 learned_addr learned my addr 172.21.15.45:0/1259077251 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 -- start start 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 librados: setting wanted keys 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 librados: calling monclient init 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient: init 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient(hunting): picked mon.noname-b con 0x7f731c073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient(hunting): picked mon.noname-a con 0x7f731c0a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient(hunting): picked mon.noname-c con 0x7f731c0a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f731c134310 con 0x7f731c0a4d20 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f731c1345f0 con 0x7f731c073d40 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f731c1348d0 con 0x7f731c0a5810 2024-09-16T03:05:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7320b75640 10 monclient(hunting): authenticate will time out at 991.291818s 2024-09-16T03:05:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7319d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f731c0a4d20 0x7f731c1de8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42278/0 (socket says 172.21.15.45:42278) 2024-09-16T03:05:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.638+0000 7f7319d9b640 1 -- 172.21.15.45:0/2381558301 learned_addr learned my addr 172.21.15.45:0/2381558301 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 client.?.objecter ms_handle_connect 0x7fce2c09d250 2024-09-16T03:05:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 client.?.objecter ms_handle_connect 0x7f731c0a4d20 2024-09-16T03:05:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 client.?.objecter ms_handle_connect 0x7fce2c0a3840 2024-09-16T03:05:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2967145591 0 0) 0x7f731c134310 con 0x7f731c0a4d20 2024-09-16T03:05:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient(hunting): my global_id is 4272 2024-09-16T03:05:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72fc004ec0 con 0x7f731c0a4d20 2024-09-16T03:05:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 client.?.objecter ms_handle_connect 0x7f731c073d40 2024-09-16T03:05:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1533052051 0 0) 0x7fce2c072e10 con 0x7fce2c09d250 2024-09-16T03:05:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): my global_id is 4288 2024-09-16T03:05:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce10004ef0 con 0x7fce2c09d250 2024-09-16T03:05:00.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fdd2c011210 con 0x7fdd340a5810 2024-09-16T03:05:00.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 client.?.objecter ms_handle_connect 0x7fce2c1d2ab0 2024-09-16T03:05:00.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 901369281 0 0) 0x7f72fc004ec0 con 0x7f731c0a4d20 2024-09-16T03:05:00.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f731c134310 con 0x7f731c0a4d20 2024-09-16T03:05:00.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7310001a20 con 0x7f731c0a4d20 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient(hunting): dump: 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.714 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.1 v1:172.21.15.46:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fdd2c04b530 con 0x7fdd340a5810 2024-09-16T03:05:00.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 10 client.4285.objecter ms_dispatch 0x7fdd34000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 3 client.4285.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 3 client.4285.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 10 client.4285.objecter checking command 1 2024-09-16T03:05:00.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 20 client.4285.objecter _get_session s=0x7fdd2004d020 osd=2 3 2024-09-16T03:05:00.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 20 client.4285.objecter put_session s=0x7fdd2004d020 osd=2 3 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 15 client.4285.objecter _session_command_op_remove -1 1 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 20 client.4285.objecter _get_session s=0x7fdd2004d020 osd=2 3 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 20 client.4285.objecter get_session s=0x7fdd2004d020 osd=2 3 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 15 client.4285.objecter _session_command_op_assign 2 1 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 20 client.4285.objecter put_session s=0x7fdd2004d020 osd=2 4 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 10 client.4285.objecter _send_command 1 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.45:6804/207876738 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd20051380 con 0x7fdd2004d6d0 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 20 client.4285.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fdd2c015ee0 con 0x7fdd340a5810 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T03:05:00.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2145120083 0 0) 0x7f731c134310 con 0x7f731c0a4d20 2024-09-16T03:05:00.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 >> v1:172.21.15.45:6790/0 conn(0x7f731c0a5810 legacy=0x7f731c1e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 >> v1:172.21.15.46:6789/0 conn(0x7f731c073d40 legacy=0x7f731c138100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 monclient: found mon.a 2024-09-16T03:05:00.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd33fff640 10 monclient: get_auth_request con 0x7fdd20046b20 auth_method 0 2024-09-16T03:05:00.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f731c072550 con 0x7f731c0a4d20 2024-09-16T03:05:00.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd337fe640 10 monclient: get_auth_request con 0x7fdd2004d6d0 auth_method 0 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 5 monclient: authenticate success, global_id 4272 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 10 monclient: _renew_subs 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f731c138810 con 0x7f731c0a4d20 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 10 monclient: _renew_subs 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f731c1d9bf0 con 0x7f731c0a4d20 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 1 librados: init done 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7310001d20 con 0x7f731c0a4d20 2024-09-16T03:05:00.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7310003ab0 con 0x7f731c0a4d20 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient: dump: 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.720 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7318d99640 4 set_mon_vals no callback set 2024-09-16T03:05:00.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 10 client.4285.objecter ms_handle_connect 0x7fdd2004d6d0 2024-09-16T03:05:00.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077130243 0 0) 0x7fce2c0a79b0 con 0x7fce2c1d2ab0 2024-09-16T03:05:00.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): my global_id is 4295 2024-09-16T03:05:00.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce2c072e10 con 0x7fce2c1d2ab0 2024-09-16T03:05:00.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 10 client.4285.objecter ms_handle_connect 0x7fdd20046b20 2024-09-16T03:05:00.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1561065548 0 0) 0x7fce10004ef0 con 0x7fce2c09d250 2024-09-16T03:05:00.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce2c0a79b0 con 0x7fce2c09d250 2024-09-16T03:05:00.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272210767 0 0) 0x7fce2c073d10 con 0x7fce2c0a3840 2024-09-16T03:05:00.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): my global_id is 4275 2024-09-16T03:05:00.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce10004ef0 con 0x7fce2c0a3840 2024-09-16T03:05:00.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 10 client.4272.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 15 client.4272.objecter _session_command_op_assign -1 1 2024-09-16T03:05:00.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 20 client.4272.objecter _get_session osd=-1 returning homeless 2024-09-16T03:05:00.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 --> v1:172.21.15.45:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7300002920 con 0x7f731c0a4d20 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fce280019a0 con 0x7fce2c09d250 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): dump: 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.724 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1568597901 0 0) 0x7fce10004ef0 con 0x7fce2c0a3840 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce2c073d10 con 0x7fce2c0a3840 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== osd.2 v1:172.21.15.45:6804/207876738 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdd20051380 con 0x7fdd2004d6d0 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 10 client.4285.objecter ms_dispatch 0x7fdd34000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 10 client.4285.objecter _finish_command 1 = system:0 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 20 client.4285.objecter put_session s=0x7fdd2004d020 osd=2 4 2024-09-16T03:05:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fdd30ff9640 15 client.4285.objecter _session_command_op_remove 2 1 2024-09-16T03:05:00.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3347942925 0 0) 0x7fce2c072e10 con 0x7fce2c1d2ab0 2024-09-16T03:05:00.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce10004ef0 con 0x7fce2c1d2ab0 2024-09-16T03:05:00.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fce1c0013c0 con 0x7fce2c0a3840 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): dump: 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.727 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fce240015a0 con 0x7fce2c1d2ab0 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient(hunting): dump: 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.728 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.729 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2158706355 0 0) 0x7fce2c0a79b0 con 0x7fce2c09d250 2024-09-16T03:05:00.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 >> v1:172.21.15.45:6790/0 conn(0x7fce2c1d2ab0 legacy=0x7fce2c1d4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 >> v1:172.21.15.45:6789/0 conn(0x7fce2c0a3840 legacy=0x7fce2c070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7310003c50 con 0x7f731c0a4d20 2024-09-16T03:05:00.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 monclient: found mon.b 2024-09-16T03:05:00.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:00.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2c071050 con 0x7fce2c09d250 2024-09-16T03:05:00.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 5 monclient: authenticate success, global_id 4288 2024-09-16T03:05:00.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 10 monclient: _renew_subs 2024-09-16T03:05:00.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:00.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fce2c072b30 con 0x7fce2c09d250 2024-09-16T03:05:00.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 10 monclient: _renew_subs 2024-09-16T03:05:00.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:00.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fce2c13dee0 con 0x7fce2c09d250 2024-09-16T03:05:00.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 1 librados: init done 2024-09-16T03:05:00.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f731004b130 con 0x7f731c0a4d20 2024-09-16T03:05:00.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 client.4272.objecter ms_dispatch 0x7f731c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 3 client.4272.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 3 client.4272.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 client.4272.objecter checking command 1 2024-09-16T03:05:00.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 20 client.4272.objecter _get_session s=0x7f72fc044430 osd=4 3 2024-09-16T03:05:00.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 20 client.4272.objecter put_session s=0x7f72fc044430 osd=4 3 2024-09-16T03:05:00.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 15 client.4272.objecter _session_command_op_remove -1 1 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 20 client.4272.objecter _get_session s=0x7f72fc044430 osd=4 3 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 20 client.4272.objecter get_session s=0x7f72fc044430 osd=4 3 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 15 client.4272.objecter _session_command_op_assign 4 1 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 20 client.4272.objecter put_session s=0x7f72fc044430 osd=4 4 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 client.4272.objecter _send_command 1 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 --> v1:172.21.15.46:6800/264199318 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72fc048790 con 0x7f72fc044ae0 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 20 client.4272.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.1 v1:172.21.15.46:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce28003620 con 0x7fce2c09d250 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fce280039f0 con 0x7fce2c09d250 2024-09-16T03:05:00.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce23fff640 4 set_mon_vals no callback set 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce227fc640 10 monclient: dump: 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.734 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f731a59c640 10 monclient: get_auth_request con 0x7f72fc03dff0 auth_method 0 2024-09-16T03:05:00.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7c49c8e640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7c49c8e640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f7c49c8e640 10 monclient: monmap: 2024-09-16T03:05:00.735 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.647599+0000 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.647599+0000 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f7310004300 con 0x7f731c0a4d20 2024-09-16T03:05:00.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T03:05:00.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f731ad9d640 10 monclient: get_auth_request con 0x7f72fc044ae0 auth_method 0 2024-09-16T03:05:00.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 10 client.4288.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 15 client.4288.objecter _session_command_op_assign -1 1 2024-09-16T03:05:00.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 20 client.4288.objecter _get_session osd=-1 returning homeless 2024-09-16T03:05:00.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:00.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.46:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fce14002920 con 0x7fce2c09d250 2024-09-16T03:05:00.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 client.4272.objecter ms_handle_connect 0x7f72fc03dff0 2024-09-16T03:05:00.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.642+0000 7f730affd640 10 client.4272.objecter ms_handle_connect 0x7f72fc044ae0 2024-09-16T03:05:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 1 Processor -- start 2024-09-16T03:05:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 1 -- start start 2024-09-16T03:05:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient: init 2024-09-16T03:05:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient(hunting): picked mon.noname-a con 0x7f7c3c007880 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient(hunting): picked mon.noname-b con 0x7f7c3c022000 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient(hunting): picked mon.noname-c con 0x7f7c3c0c1b10 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c3c01ff80 con 0x7f7c3c007880 2024-09-16T03:05:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c3c027c80 con 0x7f7c3c0c1b10 2024-09-16T03:05:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c3c01f6e0 con 0x7f7c3c022000 2024-09-16T03:05:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c43fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f7c3c022000 0x7f7c3c0c03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35070/0 (socket says 172.21.15.45:35070) 2024-09-16T03:05:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c43fff640 1 -- 172.21.15.45:0/1259482703 learned_addr learned my addr 172.21.15.45:0/1259482703 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient(hunting): authenticate will time out at 991.297144s 2024-09-16T03:05:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fce280039f0 con 0x7fce2c09d250 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.1 v1:172.21.15.46:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fce2804adb0 con 0x7fce2c09d250 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 10 client.4288.objecter ms_dispatch 0x7fce2c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 3 client.4288.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 3 client.4288.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 10 client.4288.objecter checking command 1 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 20 client.4288.objecter _get_session s=0x7fce10044a90 osd=3 3 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 20 client.4288.objecter put_session s=0x7fce10044a90 osd=3 3 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 15 client.4288.objecter _session_command_op_remove -1 1 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 20 client.4288.objecter _get_session s=0x7fce10044a90 osd=3 3 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 20 client.4288.objecter get_session s=0x7fce10044a90 osd=3 3 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 15 client.4288.objecter _session_command_op_assign 3 1 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 20 client.4288.objecter put_session s=0x7fce10044a90 osd=3 4 2024-09-16T03:05:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 10 client.4288.objecter _send_command 1 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.46:6808/3088806475 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce10048df0 con 0x7fce10045140 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 20 client.4288.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fce2804b1a0 con 0x7fce2c09d250 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce313fa640 10 monclient: get_auth_request con 0x7fce1003e540 auth_method 0 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557448659 0 0) 0x7f7c3c01f6e0 con 0x7f7c3c022000 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient(hunting): my global_id is 4298 2024-09-16T03:05:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c34003530 con 0x7f7c3c022000 2024-09-16T03:05:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce323fc640 10 monclient: get_auth_request con 0x7fce10045140 auth_method 0 2024-09-16T03:05:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278176668 0 0) 0x7f7c3c01ff80 con 0x7f7c3c007880 2024-09-16T03:05:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient(hunting): my global_id is 4278 2024-09-16T03:05:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c3c01f6e0 con 0x7f7c3c007880 2024-09-16T03:05:00.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 10 client.4288.objecter ms_handle_connect 0x7fce1003e540 2024-09-16T03:05:00.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 10 client.4288.objecter ms_handle_connect 0x7fce10045140 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1924799069 0 0) 0x7f7c34003530 con 0x7f7c3c022000 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 >> v1:172.21.15.46:6789/0 conn(0x7f7c3c0c1b10 legacy=0x7f7c3c0c3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 >> v1:172.21.15.45:6789/0 conn(0x7f7c3c007880 legacy=0x7f7c3c007c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 monclient: found mon.noname-b 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c3c0073b0 con 0x7f7c3c022000 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c300050c0 con 0x7f7c3c022000 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 5 monclient: authenticate success, global_id 4298 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: dump: 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c30005260 con 0x7f7c3c022000 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 1 -- 172.21.15.45:0/1259482703 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c30005630 con 0x7f7c3c022000 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 4 set_mon_vals no callback set 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c437fe640 10 monclient: dump: 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.748 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== osd.4 v1:172.21.15.46:6800/264199318 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72fc048790 con 0x7f72fc044ae0 2024-09-16T03:05:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f730affd640 10 client.4272.objecter ms_dispatch 0x7f731c000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:05:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f730affd640 10 client.4272.objecter _finish_command 1 = system:0 2024-09-16T03:05:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f730affd640 20 client.4272.objecter put_session s=0x7f72fc044430 osd=4 4 2024-09-16T03:05:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f730affd640 15 client.4272.objecter _session_command_op_remove 4 1 2024-09-16T03:05:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== osd.3 v1:172.21.15.46:6808/3088806475 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fce10048df0 con 0x7fce10045140 2024-09-16T03:05:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 10 client.4288.objecter ms_dispatch 0x7fce2c000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:05:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 10 client.4288.objecter _finish_command 1 = system:0 2024-09-16T03:05:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 20 client.4288.objecter put_session s=0x7fce10044a90 osd=3 4 2024-09-16T03:05:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7fce227fc640 15 client.4288.objecter _session_command_op_remove 3 1 2024-09-16T03:05:00.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 10 monclient: shutdown 2024-09-16T03:05:00.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 1 -- 172.21.15.45:0/1259482703 >> v1:172.21.15.45:6790/0 conn(0x7f7c3c022000 legacy=0x7f7c3c0c03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 1 -- 172.21.15.45:0/1259482703 shutdown_connections 2024-09-16T03:05:00.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.646+0000 7f7c49c8e640 1 -- 172.21.15.45:0/1259482703 >> 172.21.15.45:0/1259482703 conn(0x7f7c3c005e30 msgr2=0x7f7c3c006240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.650+0000 7f7c49c8e640 1 -- 172.21.15.45:0/1259482703 shutdown_connections 2024-09-16T03:05:00.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.650+0000 7f7c49c8e640 1 -- 172.21.15.45:0/1259482703 wait complete. 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient: monmap: 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 librados: starting msgr at 2024-09-16T03:05:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 librados: starting objecter 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 Processor -- start 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 -- start start 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 librados: setting wanted keys 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 librados: calling monclient init 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient: init 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient(hunting): picked mon.noname-a con 0x7f7c3c007880 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient(hunting): picked mon.noname-c con 0x7f7c3c022000 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient(hunting): picked mon.noname-b con 0x7f7c3c0c1b10 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c3c0c0d20 con 0x7f7c3c007880 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c3c0a2040 con 0x7f7c3c0c1b10 2024-09-16T03:05:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c3c1514f0 con 0x7f7c3c022000 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c48c8c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7c3c007880 0x7f7c3c1506d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42296/0 (socket says 172.21.15.45:42296) 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient(hunting): authenticate will time out at 991.306595s 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c48c8c640 1 -- 172.21.15.45:0/714189458 learned_addr learned my addr 172.21.15.45:0/714189458 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 client.?.objecter ms_handle_connect 0x7f7c3c022000 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 client.?.objecter ms_handle_connect 0x7f7c3c007880 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 client.?.objecter ms_handle_connect 0x7f7c3c0c1b10 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1731882916 0 0) 0x7f7c3c0c0d20 con 0x7f7c3c007880 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): my global_id is 4281 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c28004ec0 con 0x7f7c3c007880 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447698641 0 0) 0x7f7c3c1514f0 con 0x7f7c3c022000 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): my global_id is 4301 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c3c0c0d20 con 0x7f7c3c022000 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3738461564 0 0) 0x7f7c3c0a2040 con 0x7f7c3c0c1b10 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): my global_id is 4297 2024-09-16T03:05:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c3c1514f0 con 0x7f7c3c0c1b10 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2730180934 0 0) 0x7f7c28004ec0 con 0x7f7c3c007880 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c3c0a2040 con 0x7f7c3c007880 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c38001a80 con 0x7f7c3c007880 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): dump: 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1077854237 0 0) 0x7f7c3c0c0d20 con 0x7f7c3c022000 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c28004ec0 con 0x7f7c3c022000 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c30001120 con 0x7f7c3c022000 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient(hunting): dump: 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.760 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3060687382 0 0) 0x7f7c3c0a2040 con 0x7f7c3c007880 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 >> v1:172.21.15.45:6790/0 conn(0x7f7c3c022000 legacy=0x7f7c3c150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 >> v1:172.21.15.46:6789/0 conn(0x7f7c3c0c1b10 legacy=0x7f7c3c158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 monclient: found mon.a 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c3c0bad40 con 0x7f7c3c007880 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 5 monclient: authenticate success, global_id 4281 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient: _renew_subs 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c3c02adf0 con 0x7f7c3c007880 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient: _renew_subs 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c3c0bf430 con 0x7f7c3c007880 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.654+0000 7f7c49c8e640 1 librados: init done 2024-09-16T03:05:00.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c38001d20 con 0x7f7c3c007880 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c38003a50 con 0x7f7c3c007880 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 monclient: dump: 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.763 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c42ffd640 4 set_mon_vals no callback set 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c49c8e640 10 client.4281.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c49c8e640 15 client.4281.objecter _session_command_op_assign -1 1 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c49c8e640 20 client.4281.objecter _get_session osd=-1 returning homeless 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c49c8e640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c49c8e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7c3c022700 con 0x7f7c3c007880 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7c38003cb0 con 0x7f7c3c007880 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f7c3804b110 con 0x7f7c3c007880 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 client.4281.objecter ms_dispatch 0x7f7c44000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 3 client.4281.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 3 client.4281.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 client.4281.objecter checking command 1 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 20 client.4281.objecter _get_session s=0x7f7c280445b0 osd=1 3 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 20 client.4281.objecter put_session s=0x7f7c280445b0 osd=1 3 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 15 client.4281.objecter _session_command_op_remove -1 1 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 20 client.4281.objecter _get_session s=0x7f7c280445b0 osd=1 3 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 20 client.4281.objecter get_session s=0x7f7c280445b0 osd=1 3 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 15 client.4281.objecter _session_command_op_assign 1 1 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 20 client.4281.objecter put_session s=0x7f7c280445b0 osd=1 4 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 client.4281.objecter _send_command 1 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6808/3228817795 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c28048910 con 0x7f7c28044c60 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 20 client.4281.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f7c3804b520 con 0x7f7c3c007880 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c4aef0640 10 monclient: get_auth_request con 0x7f7c28044c60 auth_method 0 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c43fff640 10 monclient: get_auth_request con 0x7f7c2803e080 auth_method 0 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 client.4281.objecter ms_handle_connect 0x7f7c28044c60 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 client.4281.objecter ms_handle_connect 0x7f7c2803e080 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 10 client.4285.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 15 client.4285.objecter _session_command_op_assign -1 2 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 20 client.4285.objecter _get_session s=0x7fdd2004d020 osd=2 3 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 20 client.4285.objecter put_session s=0x7fdd2004d020 osd=2 3 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 20 client.4285.objecter _get_session s=0x7fdd2004d020 osd=2 3 2024-09-16T03:05:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 15 client.4285.objecter _session_command_op_remove -1 2 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 20 client.4285.objecter get_session s=0x7fdd2004d020 osd=2 3 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 15 client.4285.objecter _session_command_op_assign 2 2 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 20 client.4285.objecter put_session s=0x7fdd2004d020 osd=2 4 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 10 client.4285.objecter _send_command 2 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 --> v1:172.21.15.45:6804/207876738 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdd14001230 con 0x7fdd2004d6d0 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== osd.1 v1:172.21.15.45:6808/3228817795 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7c28048910 con 0x7f7c28044c60 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 client.4281.objecter ms_dispatch 0x7f7c44000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 10 client.4281.objecter _finish_command 1 = system:0 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 20 client.4281.objecter put_session s=0x7f7c280445b0 osd=1 4 2024-09-16T03:05:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7f7c417fa640 15 client.4281.objecter _session_command_op_remove 1 1 2024-09-16T03:05:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd30ff9640 1 -- 172.21.15.45:0/2771466812 <== osd.2 v1:172.21.15.45:6804/207876738 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7fdd14001230 con 0x7fdd2004d6d0 2024-09-16T03:05:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd30ff9640 10 client.4285.objecter ms_dispatch 0x7fdd34000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:05:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd30ff9640 10 client.4285.objecter _finish_command 2 = system:0 2024-09-16T03:05:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd30ff9640 20 client.4285.objecter put_session s=0x7fdd2004d020 osd=2 4 2024-09-16T03:05:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd30ff9640 15 client.4285.objecter _session_command_op_remove 2 2 2024-09-16T03:05:00.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 10 librados: watch_flush enter 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 10 librados: watch_flush exit 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 10 client.4285.objecter close_session for osd.2 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 >> v1:172.21.15.45:6804/207876738 conn(0x7fdd2004d6d0 legacy=0x7fdd2004fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 20 client.4285.objecter put_session s=0x7fdd2004d020 osd=2 1 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 20 client.4285.objecter shutdown clearing up homeless session... 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 10 client.4285.objecter successfully canceled tick 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 >> v1:172.21.15.46:6812/149360280 conn(0x7fdd20046b20 legacy=0x7fdd20048fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 10 monclient: shutdown 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 >> v1:172.21.15.46:6789/0 conn(0x7fdd340a5810 legacy=0x7fdd341d51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd388e6640 1 -- 172.21.15.45:0/2771466812 reap_dead start 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 shutdown_connections 2024-09-16T03:05:00.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.658+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 >> 172.21.15.45:0/2771466812 conn(0x7fdd340aa720 msgr2=0x7fdd341415f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 shutdown_connections 2024-09-16T03:05:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fdd3a349640 1 -- 172.21.15.45:0/2771466812 wait complete. 2024-09-16T03:05:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fdd3a349640 1 librados: shutdown 2024-09-16T03:05:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 10 client.4288.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:05:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 15 client.4288.objecter _session_command_op_assign -1 2 2024-09-16T03:05:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 20 client.4288.objecter _get_session s=0x7fce10044a90 osd=3 3 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 20 client.4288.objecter put_session s=0x7fce10044a90 osd=3 3 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 20 client.4288.objecter _get_session s=0x7fce10044a90 osd=3 3 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 15 client.4288.objecter _session_command_op_remove -1 2 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 20 client.4288.objecter get_session s=0x7fce10044a90 osd=3 3 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 15 client.4288.objecter _session_command_op_assign 3 2 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 20 client.4288.objecter put_session s=0x7fce10044a90 osd=3 4 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 10 client.4288.objecter _send_command 2 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 --> v1:172.21.15.46:6808/3088806475 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fce14001250 con 0x7fce10045140 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce227fc640 1 -- 172.21.15.45:0/1259077251 <== osd.3 v1:172.21.15.46:6808/3088806475 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7fce14001250 con 0x7fce10045140 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce227fc640 10 client.4288.objecter ms_dispatch 0x7fce2c000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce227fc640 10 client.4288.objecter _finish_command 2 = system:0 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce227fc640 20 client.4288.objecter put_session s=0x7fce10044a90 osd=3 4 2024-09-16T03:05:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.662+0000 7fce227fc640 15 client.4288.objecter _session_command_op_remove 3 2 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 10 librados: watch_flush enter 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 10 librados: watch_flush exit 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 10 client.4288.objecter close_session for osd.3 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 >> v1:172.21.15.46:6808/3088806475 conn(0x7fce10045140 legacy=0x7fce10047580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 20 client.4288.objecter put_session s=0x7fce10044a90 osd=3 1 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 20 client.4288.objecter shutdown clearing up homeless session... 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 10 client.4288.objecter successfully canceled tick 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 >> v1:172.21.15.46:6812/149360280 conn(0x7fce1003e540 legacy=0x7fce10040a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 10 monclient: shutdown 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 >> v1:172.21.15.46:6789/0 conn(0x7fce2c09d250 legacy=0x7fce2c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce323fc640 1 -- 172.21.15.45:0/1259077251 reap_dead start 2024-09-16T03:05:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 shutdown_connections 2024-09-16T03:05:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 >> 172.21.15.45:0/1259077251 conn(0x7fce2c0a5310 msgr2=0x7fce2c0a5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 shutdown_connections 2024-09-16T03:05:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 1 -- 172.21.15.45:0/1259077251 wait complete. 2024-09-16T03:05:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.666+0000 7fce32bfd640 1 librados: shutdown 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 10 client.4272.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 15 client.4272.objecter _session_command_op_assign -1 2 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 20 client.4272.objecter _get_session s=0x7f72fc044430 osd=4 3 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 20 client.4272.objecter put_session s=0x7f72fc044430 osd=4 3 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 20 client.4272.objecter _get_session s=0x7f72fc044430 osd=4 3 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 15 client.4272.objecter _session_command_op_remove -1 2 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 20 client.4272.objecter get_session s=0x7f72fc044430 osd=4 3 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 15 client.4272.objecter _session_command_op_assign 4 2 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 20 client.4272.objecter put_session s=0x7f72fc044430 osd=4 4 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 10 client.4272.objecter _send_command 2 2024-09-16T03:05:00.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 --> v1:172.21.15.46:6800/264199318 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7300001230 con 0x7f72fc044ae0 2024-09-16T03:05:00.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f730affd640 1 -- 172.21.15.45:0/2381558301 <== osd.4 v1:172.21.15.46:6800/264199318 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3543570501) 0x7f7300001230 con 0x7f72fc044ae0 2024-09-16T03:05:00.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f730affd640 10 client.4272.objecter ms_dispatch 0x7f731c000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:05:00.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f730affd640 10 client.4272.objecter _finish_command 2 = system:0 2024-09-16T03:05:00.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f730affd640 20 client.4272.objecter put_session s=0x7f72fc044430 osd=4 4 2024-09-16T03:05:00.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f730affd640 15 client.4272.objecter _session_command_op_remove 4 2 2024-09-16T03:05:00.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 10 librados: watch_flush enter 2024-09-16T03:05:00.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 10 librados: watch_flush exit 2024-09-16T03:05:00.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 10 client.4272.objecter close_session for osd.4 2024-09-16T03:05:00.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 >> v1:172.21.15.46:6800/264199318 conn(0x7f72fc044ae0 legacy=0x7f72fc046f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 20 client.4272.objecter put_session s=0x7f72fc044430 osd=4 1 2024-09-16T03:05:00.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 20 client.4272.objecter shutdown clearing up homeless session... 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 10 client.4272.objecter successfully canceled tick 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 >> v1:172.21.15.46:6812/149360280 conn(0x7f72fc03dff0 legacy=0x7f72fc0404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 10 monclient: shutdown 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 >> v1:172.21.15.45:6789/0 conn(0x7f731c0a4d20 legacy=0x7f731c1de8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f731ad9d640 1 -- 172.21.15.45:0/2381558301 reap_dead start 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 shutdown_connections 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 >> 172.21.15.45:0/2381558301 conn(0x7f731c0aa720 msgr2=0x7f731c0a97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 shutdown_connections 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 1 -- 172.21.15.45:0/2381558301 wait complete. 2024-09-16T03:05:00.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.670+0000 7f7320b75640 1 librados: shutdown 2024-09-16T03:05:00.780 INFO:teuthology.orchestra.run.smithi045.stdout:47244640260 2024-09-16T03:05:00.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:05:00.781 INFO:teuthology.orchestra.run.smithi045.stdout:51539607556 2024-09-16T03:05:00.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:05:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 10 client.4281.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:05:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 15 client.4281.objecter _session_command_op_assign -1 2 2024-09-16T03:05:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 20 client.4281.objecter _get_session s=0x7f7c280445b0 osd=1 3 2024-09-16T03:05:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 20 client.4281.objecter put_session s=0x7f7c280445b0 osd=1 3 2024-09-16T03:05:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 20 client.4281.objecter _get_session s=0x7f7c280445b0 osd=1 3 2024-09-16T03:05:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 15 client.4281.objecter _session_command_op_remove -1 2 2024-09-16T03:05:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 20 client.4281.objecter get_session s=0x7f7c280445b0 osd=1 3 2024-09-16T03:05:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 15 client.4281.objecter _session_command_op_assign 1 2 2024-09-16T03:05:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 20 client.4281.objecter put_session s=0x7f7c280445b0 osd=1 4 2024-09-16T03:05:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 10 client.4281.objecter _send_command 2 2024-09-16T03:05:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 --> v1:172.21.15.45:6808/3228817795 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c3c0c55a0 con 0x7f7c28044c60 2024-09-16T03:05:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c417fa640 1 -- 172.21.15.45:0/714189458 <== osd.1 v1:172.21.15.45:6808/3228817795 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7f7c3c0c55a0 con 0x7f7c28044c60 2024-09-16T03:05:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c417fa640 10 client.4281.objecter ms_dispatch 0x7f7c44000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:05:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c417fa640 10 client.4281.objecter _finish_command 2 = system:0 2024-09-16T03:05:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c417fa640 20 client.4281.objecter put_session s=0x7f7c280445b0 osd=1 4 2024-09-16T03:05:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c417fa640 15 client.4281.objecter _session_command_op_remove 1 2 2024-09-16T03:05:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 10 librados: watch_flush enter 2024-09-16T03:05:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 10 librados: watch_flush exit 2024-09-16T03:05:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 10 client.4281.objecter close_session for osd.1 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 >> v1:172.21.15.45:6808/3228817795 conn(0x7f7c28044c60 legacy=0x7f7c280470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 20 client.4281.objecter put_session s=0x7f7c280445b0 osd=1 1 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 20 client.4281.objecter shutdown clearing up homeless session... 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 10 client.4281.objecter successfully canceled tick 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 >> v1:172.21.15.46:6812/149360280 conn(0x7f7c2803e080 legacy=0x7f7c28040540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 10 monclient: shutdown 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 >> v1:172.21.15.45:6789/0 conn(0x7f7c3c007880 legacy=0x7f7c3c1506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c4aef0640 1 -- 172.21.15.45:0/714189458 reap_dead start 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 shutdown_connections 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 >> 172.21.15.45:0/714189458 conn(0x7f7c3c005e30 msgr2=0x7f7c3c022930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 shutdown_connections 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 1 -- 172.21.15.45:0/714189458 wait complete. 2024-09-16T03:05:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.686+0000 7f7c49c8e640 1 librados: shutdown 2024-09-16T03:05:00.786 INFO:teuthology.orchestra.run.smithi045.stdout:51539607557 2024-09-16T03:05:00.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:05:00.787 INFO:teuthology.orchestra.run.smithi045.stdout:51539607556 2024-09-16T03:05:00.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient: monmap: 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.869512+0000 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.869512+0000 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.871 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.871 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 1 Processor -- start 2024-09-16T03:05:00.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 1 -- start start 2024-09-16T03:05:00.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient: init 2024-09-16T03:05:00.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:05:00.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient(hunting): picked mon.noname-b con 0x7f249c0a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient(hunting): picked mon.noname-a con 0x7f249c0a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient(hunting): picked mon.noname-c con 0x7f249c073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c09d9e0 con 0x7f249c0a5810 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f249c0a5810 0x7f249c13cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42310/0 (socket says 172.21.15.45:42310) 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249affd640 1 -- 172.21.15.45:0/4208896201 learned_addr learned my addr 172.21.15.45:0/4208896201 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 1 -- 172.21.15.45:0/4208896201 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c09dc20 con 0x7f249c073d40 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 1 -- 172.21.15.45:0/4208896201 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c073360 con 0x7f249c0a4d20 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f24a2138640 10 monclient(hunting): authenticate will time out at 991.519568s 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3882795653 0 0) 0x7f249c073360 con 0x7f249c0a4d20 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): my global_id is 4304 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2484003530 con 0x7f249c0a4d20 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330238100 0 0) 0x7f249c09d9e0 con 0x7f249c0a5810 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): my global_id is 4284 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f249c073360 con 0x7f249c0a5810 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565409049 0 0) 0x7f249c09dc20 con 0x7f249c073d40 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 10 monclient(hunting): my global_id is 4300 2024-09-16T03:05:00.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.866+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f249c09d9e0 con 0x7f249c073d40 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1097849356 0 0) 0x7f2484003530 con 0x7f249c0a4d20 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 >> v1:172.21.15.46:6789/0 conn(0x7f249c073d40 legacy=0x7f249c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 >> v1:172.21.15.45:6789/0 conn(0x7f249c0a5810 legacy=0x7f249c13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 1 monclient: found mon.noname-b 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c09cf30 con 0x7f249c0a4d20 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f248c0016a0 con 0x7f249c0a4d20 2024-09-16T03:05:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: dump: 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 5 monclient: authenticate success, global_id 4304 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f248c001840 con 0x7f249c0a4d20 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 1 -- 172.21.15.45:0/4208896201 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f248c001cd0 con 0x7f249c0a4d20 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f249a7fc640 10 monclient: dump: 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.879 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.879 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.879 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.879 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 4 set_mon_vals no callback set 2024-09-16T03:05:00.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 10 monclient: shutdown 2024-09-16T03:05:00.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 1 -- 172.21.15.45:0/4208896201 >> v1:172.21.15.45:6790/0 conn(0x7f249c0a4d20 legacy=0x7f249c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 1 -- 172.21.15.45:0/4208896201 shutdown_connections 2024-09-16T03:05:00.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 1 -- 172.21.15.45:0/4208896201 >> 172.21.15.45:0/4208896201 conn(0x7f249c0aa720 msgr2=0x7f249c0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 1 -- 172.21.15.45:0/4208896201 shutdown_connections 2024-09-16T03:05:00.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 1 -- 172.21.15.45:0/4208896201 wait complete. 2024-09-16T03:05:00.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.870+0000 7f24a2138640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 10 monclient: monmap: 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.884 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 1 librados: starting msgr at 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 1 librados: starting objecter 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 1 Processor -- start 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 1 -- start start 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 1 librados: setting wanted keys 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 1 librados: calling monclient init 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 10 monclient: init 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 10 monclient(hunting): picked mon.noname-a con 0x7f249c073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 10 monclient(hunting): picked mon.noname-c con 0x7f249c0a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f249affd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f249c0a4d20 0x7f249c13cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35094/0 (socket says 172.21.15.45:35094) 2024-09-16T03:05:00.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f249b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f249c073d40 0x7f249c13c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42324/0 (socket says 172.21.15.45:42324) 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f249affd640 1 -- 172.21.15.45:0/1568517313 learned_addr learned my addr 172.21.15.45:0/1568517313 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 client.?.objecter ms_handle_connect 0x7f249c073d40 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 client.?.objecter ms_handle_connect 0x7f249c0a4d20 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 10 monclient(hunting): picked mon.noname-b con 0x7f249c0a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c140520 con 0x7f249c073d40 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c13d650 con 0x7f249c0a5810 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c0a1510 con 0x7f249c0a4d20 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f24a2138640 10 monclient(hunting): authenticate will time out at 991.527264s 2024-09-16T03:05:00.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236453189 0 0) 0x7f249c140520 con 0x7f249c073d40 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): my global_id is 4287 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2488004ec0 con 0x7f249c073d40 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897620379 0 0) 0x7f249c0a1510 con 0x7f249c0a4d20 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): my global_id is 4307 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f249c140520 con 0x7f249c0a4d20 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 client.?.objecter ms_handle_connect 0x7f249c0a5810 2024-09-16T03:05:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 756803895 0 0) 0x7f2488004ec0 con 0x7f249c073d40 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f249c0a1510 con 0x7f249c073d40 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f248c001c50 con 0x7f249c073d40 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): dump: 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.888 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.874+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 352068148 0 0) 0x7f249c140520 con 0x7f249c0a4d20 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2488004ec0 con 0x7f249c0a4d20 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050792489 0 0) 0x7f249c13d650 con 0x7f249c0a5810 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient(hunting): my global_id is 4303 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f249c140520 con 0x7f249c0a5810 2024-09-16T03:05:00.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2514397345 0 0) 0x7f249c0a1510 con 0x7f249c073d40 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 >> v1:172.21.15.45:6790/0 conn(0x7f249c0a4d20 legacy=0x7f249c13cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 >> v1:172.21.15.46:6789/0 conn(0x7f249c0a5810 legacy=0x7f249c1d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 monclient: found mon.a 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c13f6b0 con 0x7f249c073d40 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 5 monclient: authenticate success, global_id 4287 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 10 monclient: _renew_subs 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f249c13e1c0 con 0x7f249c073d40 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 10 monclient: _renew_subs 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f249c1388e0 con 0x7f249c073d40 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 1 librados: init done 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f249c13f150 con 0x7f249c073d40 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f248c0017a0 con 0x7f249c073d40 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f248c003b10 con 0x7f249c073d40 2024-09-16T03:05:00.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 10 monclient: dump: 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f248c003d70 con 0x7f249c073d40 2024-09-16T03:05:00.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f2499ffb640 4 set_mon_vals no callback set 2024-09-16T03:05:00.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.878+0000 7f249affd640 10 monclient: get_auth_request con 0x7f248803e190 auth_method 0 2024-09-16T03:05:00.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.882+0000 7f2483fff640 10 client.4287.objecter ms_handle_connect 0x7f248803e190 2024-09-16T03:05:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 10 monclient: monmap: 2024-09-16T03:05:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.890870+0000 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.890870+0000 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 1 Processor -- start 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 1 -- start start 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 10 monclient: init 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d380a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.886+0000 7f2d3e00d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d380a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d38073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3809d9e0 con 0x7f2d380a4d20 2024-09-16T03:05:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3809dc20 con 0x7f2d380a5810 2024-09-16T03:05:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d37fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2d38073d40 0x7f2d38140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35106/0 (socket says 172.21.15.45:35106) 2024-09-16T03:05:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d37fff640 1 -- 172.21.15.45:0/3475834049 learned_addr learned my addr 172.21.15.45:0/3475834049 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d377fe640 1 --1- 172.21.15.45:0/3475834049 >> v1:172.21.15.45:6789/0 conn(0x7f2d380a4d20 0x7f2d380a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42338/0 (socket says 172.21.15.45:42338) 2024-09-16T03:05:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f248c049e40 con 0x7f249c073d40 2024-09-16T03:05:00.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2483fff640 10 client.4287.objecter ms_dispatch 0x7f249c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2483fff640 3 client.4287.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2483fff640 3 client.4287.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2483fff640 20 client.4287.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3475834049 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d38073360 con 0x7f2d38073d40 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 10 monclient(hunting): authenticate will time out at 991.540690s 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83414609 0 0) 0x7f2d38073360 con 0x7f2d38073d40 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient(hunting): my global_id is 4310 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d20003530 con 0x7f2d38073d40 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3221718553 0 0) 0x7f2d3809dc20 con 0x7f2d380a5810 2024-09-16T03:05:00.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient(hunting): my global_id is 4306 2024-09-16T03:05:00.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d38073360 con 0x7f2d380a5810 2024-09-16T03:05:00.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f249c13f150 con 0x7f249c073d40 2024-09-16T03:05:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2483fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2483fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:00.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3090617305 0 0) 0x7f2d20003530 con 0x7f2d38073d40 2024-09-16T03:05:00.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 >> v1:172.21.15.46:6789/0 conn(0x7f2d380a5810 legacy=0x7f2d3813cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 >> v1:172.21.15.45:6789/0 conn(0x7f2d380a4d20 legacy=0x7f2d380a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 monclient: found mon.noname-b 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3809cf30 con 0x7f2d38073d40 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d30001950 con 0x7f2d38073d40 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: dump: 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d30001af0 con 0x7f2d38073d40 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 1 -- 172.21.15.45:0/3475834049 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d30001f80 con 0x7f2d38073d40 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d367fc640 10 monclient: dump: 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.904 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 5 monclient: authenticate success, global_id 4310 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 4 set_mon_vals no callback set 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 10 monclient: shutdown 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3475834049 >> v1:172.21.15.45:6790/0 conn(0x7f2d38073d40 legacy=0x7f2d38140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3475834049 shutdown_connections 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3475834049 >> 172.21.15.45:0/3475834049 conn(0x7f2d380aa720 msgr2=0x7f2d380aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3475834049 shutdown_connections 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3475834049 wait complete. 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.890+0000 7f2d3e00d640 10 monclient: monmap: 2024-09-16T03:05:00.905 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 librados: starting msgr at 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 librados: starting objecter 2024-09-16T03:05:00.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 Processor -- start 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 -- start start 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 librados: setting wanted keys 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 librados: calling monclient init 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient: init 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d38073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d380a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d380a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d36ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2d380a4d20 0x7f2d3809e940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42354/0 (socket says 172.21.15.45:42354) 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d36ffd640 1 -- 172.21.15.45:0/3105839754 learned_addr learned my addr 172.21.15.45:0/3105839754 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 client.?.objecter ms_handle_connect 0x7f2d380a4d20 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3809f050 con 0x7f2d380a4d20 2024-09-16T03:05:00.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 client.?.objecter ms_handle_connect 0x7f2d38073d40 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 client.?.objecter ms_handle_connect 0x7f2d380a5810 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d38120f30 con 0x7f2d380a5810 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d380a1dc0 con 0x7f2d38073d40 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient(hunting): authenticate will time out at 991.545636s 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1367164094 0 0) 0x7f2d3809f050 con 0x7f2d380a4d20 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): my global_id is 4293 2024-09-16T03:05:00.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d24004ec0 con 0x7f2d380a4d20 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292352694 0 0) 0x7f2d38120f30 con 0x7f2d380a5810 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): my global_id is 4309 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d3809f050 con 0x7f2d380a5810 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1823737178 0 0) 0x7f2d380a1dc0 con 0x7f2d38073d40 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): my global_id is 4313 2024-09-16T03:05:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d38120f30 con 0x7f2d38073d40 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1021763098 0 0) 0x7f2d24004ec0 con 0x7f2d380a4d20 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d380a1dc0 con 0x7f2d380a4d20 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d2c001a20 con 0x7f2d380a4d20 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): dump: 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.910 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1841650311 0 0) 0x7f2d380a1dc0 con 0x7f2d380a4d20 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 >> v1:172.21.15.45:6790/0 conn(0x7f2d38073d40 legacy=0x7f2d3809e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 >> v1:172.21.15.46:6789/0 conn(0x7f2d380a5810 legacy=0x7f2d381f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 monclient: found mon.a 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d380a1a00 con 0x7f2d380a4d20 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 5 monclient: authenticate success, global_id 4293 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient: _renew_subs 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2d380a1510 con 0x7f2d380a4d20 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient: _renew_subs 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2d38120c50 con 0x7f2d380a4d20 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 librados: init done 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d2c001d20 con 0x7f2d380a4d20 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d2c003a70 con 0x7f2d380a4d20 2024-09-16T03:05:00.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 10 monclient: dump: 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d35ffb640 4 set_mon_vals no callback set 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f2d2c003c10 con 0x7f2d380a4d20 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.894+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d18005090 con 0x7f2d380a4d20 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.898+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f2d2c04b1a0 con 0x7f2d380a4d20 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.898+0000 7f2d1ffff640 10 client.4293.objecter ms_dispatch 0x7f2d38000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.898+0000 7f2d1ffff640 3 client.4293.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.898+0000 7f2d1ffff640 3 client.4293.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.898+0000 7f2d1ffff640 20 client.4293.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.898+0000 7f2d377fe640 10 monclient: get_auth_request con 0x7f2d24081f40 auth_method 0 2024-09-16T03:05:00.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.898+0000 7f2d1ffff640 10 client.4293.objecter ms_handle_connect 0x7f2d24081f40 2024-09-16T03:05:00.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.906+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f2d2c016ba0 con 0x7f2d380a4d20 2024-09-16T03:05:00.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.906+0000 7f2d1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.906+0000 7f2d1ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.938+0000 7fe4a410e640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.938+0000 7fe4a410e640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.938+0000 7fe4a410e640 10 monclient: monmap: 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.944037+0000 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.944037+0000 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.945 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 1 Processor -- start 2024-09-16T03:05:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 1 -- start start 2024-09-16T03:05:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient: init 2024-09-16T03:05:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient(hunting): picked mon.noname-c con 0x7fe49c0a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient(hunting): picked mon.noname-b con 0x7fe49c0a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a16a9640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe49c0a5810 0x7fe49c13cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35124/0 (socket says 172.21.15.45:35124) 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a16a9640 1 -- 172.21.15.45:0/1771109418 learned_addr learned my addr 172.21.15.45:0/1771109418 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient(hunting): picked mon.noname-a con 0x7fe49c073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 1 -- 172.21.15.45:0/1771109418 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe49c09d9e0 con 0x7fe49c073d40 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 1 -- 172.21.15.45:0/1771109418 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe49c09dc20 con 0x7fe49c0a4d20 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 1 -- 172.21.15.45:0/1771109418 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe49c073360 con 0x7fe49c0a5810 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient(hunting): authenticate will time out at 991.593855s 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550867182 0 0) 0x7fe49c09dc20 con 0x7fe49c0a4d20 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): my global_id is 4312 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe488003530 con 0x7fe49c0a4d20 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031164668 0 0) 0x7fe49c073360 con 0x7fe49c0a5810 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): my global_id is 4316 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe49c09dc20 con 0x7fe49c0a5810 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3778809403 0 0) 0x7fe49c09d9e0 con 0x7fe49c073d40 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient(hunting): my global_id is 4296 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe49c073360 con 0x7fe49c073d40 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1487659995 0 0) 0x7fe488003530 con 0x7fe49c0a4d20 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 >> v1:172.21.15.45:6790/0 conn(0x7fe49c0a5810 legacy=0x7fe49c13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 >> v1:172.21.15.45:6789/0 conn(0x7fe49c073d40 legacy=0x7fe49c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 monclient: found mon.noname-c 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49c09cf30 con 0x7fe49c0a4d20 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe498001620 con 0x7fe49c0a4d20 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: dump: 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 5 monclient: authenticate success, global_id 4312 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe498001880 con 0x7fe49c0a4d20 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 1 -- 172.21.15.45:0/1771109418 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe498001c50 con 0x7fe49c0a4d20 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 4 set_mon_vals no callback set 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a0ea8640 10 monclient: dump: 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 10 monclient: shutdown 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.942+0000 7fe4a410e640 1 -- 172.21.15.45:0/1771109418 >> v1:172.21.15.46:6789/0 conn(0x7fe49c0a4d20 legacy=0x7fe49c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 -- 172.21.15.45:0/1771109418 shutdown_connections 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 -- 172.21.15.45:0/1771109418 >> 172.21.15.45:0/1771109418 conn(0x7fe49c0aa720 msgr2=0x7fe49c0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 -- 172.21.15.45:0/1771109418 shutdown_connections 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 -- 172.21.15.45:0/1771109418 wait complete. 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient: monmap: 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.956 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 librados: starting msgr at 2024-09-16T03:05:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 librados: starting objecter 2024-09-16T03:05:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 Processor -- start 2024-09-16T03:05:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 -- start start 2024-09-16T03:05:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 librados: setting wanted keys 2024-09-16T03:05:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 librados: calling monclient init 2024-09-16T03:05:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient: init 2024-09-16T03:05:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient(hunting): picked mon.noname-b con 0x7fe49c073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient(hunting): picked mon.noname-c con 0x7fe49c0a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient(hunting): picked mon.noname-a con 0x7fe49c0a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a26ab640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe49c0a5810 0x7fe49c1d5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42370/0 (socket says 172.21.15.45:42370) 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a26ab640 1 -- 172.21.15.45:0/2499095044 learned_addr learned my addr 172.21.15.45:0/2499095044 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 client.?.objecter ms_handle_connect 0x7fe49c0a5810 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 client.?.objecter ms_handle_connect 0x7fe49c0a4d20 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 client.?.objecter ms_handle_connect 0x7fe49c073d40 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe49c140a40 con 0x7fe49c0a5810 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe49c13da40 con 0x7fe49c073d40 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe49c13fbd0 con 0x7fe49c0a4d20 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677085591 0 0) 0x7fe49c140a40 con 0x7fe49c0a5810 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4a410e640 10 monclient(hunting): authenticate will time out at 991.599301s 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): my global_id is 4299 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe480004ec0 con 0x7fe49c0a5810 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130852137 0 0) 0x7fe49c13fbd0 con 0x7fe49c0a4d20 2024-09-16T03:05:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): my global_id is 4319 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe49c140a40 con 0x7fe49c0a4d20 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3573348574 0 0) 0x7fe49c13da40 con 0x7fe49c073d40 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): my global_id is 4315 2024-09-16T03:05:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe49c13fbd0 con 0x7fe49c073d40 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7f9dfaf33640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1720314147 0 0) 0x7fe480004ec0 con 0x7fe49c0a5810 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe49c13da40 con 0x7fe49c0a5810 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe494001d20 con 0x7fe49c0a5810 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): dump: 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.964 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7f9dfaf33640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7f9dfaf33640 10 monclient: monmap: 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.951912+0000 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.951912+0000 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.965 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.946+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 499030315 0 0) 0x7fe49c140a40 con 0x7fe49c0a4d20 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe480004ec0 con 0x7fe49c0a4d20 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe48c001980 con 0x7fe49c0a4d20 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient(hunting): dump: 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3054899161 0 0) 0x7fe49c13fbd0 con 0x7fe49c073d40 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe49c140a40 con 0x7fe49c073d40 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe498001560 con 0x7fe49c073d40 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient(hunting): dump: 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 893648799 0 0) 0x7fe49c13da40 con 0x7fe49c0a5810 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 >> v1:172.21.15.45:6790/0 conn(0x7fe49c0a4d20 legacy=0x7fe49c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 >> v1:172.21.15.46:6789/0 conn(0x7fe49c073d40 legacy=0x7fe49c13cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 monclient: found mon.a 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49c13e6e0 con 0x7fe49c0a5810 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 5 monclient: authenticate success, global_id 4299 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 10 monclient: _renew_subs 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe49c140760 con 0x7fe49c0a5810 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 10 monclient: _renew_subs 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe49c120970 con 0x7fe49c0a5810 2024-09-16T03:05:00.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 1 librados: init done 2024-09-16T03:05:00.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 1 Processor -- start 2024-09-16T03:05:00.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 1 -- start start 2024-09-16T03:05:00.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 10 monclient: init 2024-09-16T03:05:00.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:00.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe49c138ae0 con 0x7fe49c0a5810 2024-09-16T03:05:00.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:00.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 10 monclient(hunting): picked mon.noname-c con 0x7f9df40a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe4940039a0 con 0x7fe49c0a5810 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe494003d70 con 0x7fe49c0a5810 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 monclient: dump: 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.973 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 10 monclient(hunting): picked mon.noname-a con 0x7f9df40a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 10 monclient(hunting): picked mon.noname-b con 0x7f9df4073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df409d9e0 con 0x7f9df40a5810 2024-09-16T03:05:00.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df409dc20 con 0x7f9df40a4d20 2024-09-16T03:05:00.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df94d0640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9df4073d40 0x7f9df4140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35132/0 (socket says 172.21.15.45:35132) 2024-09-16T03:05:00.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df94d0640 1 -- 172.21.15.45:0/2681206272 learned_addr learned my addr 172.21.15.45:0/2681206272 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fe494003fd0 con 0x7fe49c0a5810 2024-09-16T03:05:00.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe493fff640 4 set_mon_vals no callback set 2024-09-16T03:05:00.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 1 -- 172.21.15.45:0/2681206272 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df4073360 con 0x7f9df4073d40 2024-09-16T03:05:00.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 10 monclient(hunting): authenticate will time out at 991.602436s 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911881862 0 0) 0x7f9df4073360 con 0x7f9df4073d40 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient(hunting): my global_id is 4322 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de4003530 con 0x7f9df4073d40 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 203135065 0 0) 0x7f9df409dc20 con 0x7f9df40a4d20 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient(hunting): my global_id is 4318 2024-09-16T03:05:00.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9df4073360 con 0x7f9df40a4d20 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 116799184 0 0) 0x7f9de4003530 con 0x7f9df4073d40 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 >> v1:172.21.15.46:6789/0 conn(0x7f9df40a4d20 legacy=0x7f9df40a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 >> v1:172.21.15.45:6789/0 conn(0x7f9df40a5810 legacy=0x7f9df413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 monclient: found mon.noname-b 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df409cf30 con 0x7f9df4073d40 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 5 monclient: authenticate success, global_id 4322 2024-09-16T03:05:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9dec001a20 con 0x7f9df4073d40 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: dump: 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9dec001c40 con 0x7f9df4073d40 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 1 -- 172.21.15.45:0/2681206272 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9dec002090 con 0x7f9df4073d40 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9df37fe640 10 monclient: dump: 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.979 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.980 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.980 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.980 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.980 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:00.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7f9dfaf33640 4 set_mon_vals no callback set 2024-09-16T03:05:00.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fe49404b260 con 0x7fe49c0a5810 2024-09-16T03:05:00.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 10 client.4299.objecter ms_dispatch 0x7fe49c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 3 client.4299.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 3 client.4299.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4927fc640 20 client.4299.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.950+0000 7fe4a1eaa640 10 monclient: get_auth_request con 0x7fe48003e4a0 auth_method 0 2024-09-16T03:05:00.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.954+0000 7fe4927fc640 10 client.4299.objecter ms_handle_connect 0x7fe48003e4a0 2024-09-16T03:05:00.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.954+0000 7f9dfaf33640 10 monclient: shutdown 2024-09-16T03:05:00.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.954+0000 7f9dfaf33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:00.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.954+0000 7f9dfaf33640 1 -- 172.21.15.45:0/2681206272 >> v1:172.21.15.45:6790/0 conn(0x7f9df4073d40 legacy=0x7f9df4140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.954+0000 7f9dfaf33640 1 -- 172.21.15.45:0/2681206272 shutdown_connections 2024-09-16T03:05:00.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 -- 172.21.15.45:0/2681206272 >> 172.21.15.45:0/2681206272 conn(0x7f9df40aa720 msgr2=0x7f9df40aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:00.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fe494016c00 con 0x7fe49c0a5810 2024-09-16T03:05:00.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7fe4927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7fe4927fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:00.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 -- 172.21.15.45:0/2681206272 shutdown_connections 2024-09-16T03:05:00.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 -- 172.21.15.45:0/2681206272 wait complete. 2024-09-16T03:05:00.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient: monmap: 2024-09-16T03:05:00.985 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.985 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:00.985 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 librados: starting msgr at 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 librados: starting objecter 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 Processor -- start 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 -- start start 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 librados: setting wanted keys 2024-09-16T03:05:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 librados: calling monclient init 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient: init 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient(hunting): picked mon.noname-c con 0x7f9df4073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient(hunting): picked mon.noname-b con 0x7f9df40a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient(hunting): picked mon.noname-a con 0x7f9df40a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df40a78f0 con 0x7f9df40a5810 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df41403b0 con 0x7f9df40a4d20 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df4140690 con 0x7f9df4073d40 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9dfaf33640 10 monclient(hunting): authenticate will time out at 991.610704s 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df8ccf640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9df4073d40 0x7f9df41cd1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35146/0 (socket says 172.21.15.45:35146) 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df8ccf640 1 -- 172.21.15.45:0/3165632713 learned_addr learned my addr 172.21.15.45:0/3165632713 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 client.?.objecter ms_handle_connect 0x7f9df4073d40 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1428573078 0 0) 0x7f9df4140690 con 0x7f9df4073d40 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): my global_id is 4325 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dd8004ec0 con 0x7f9df4073d40 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 client.?.objecter ms_handle_connect 0x7f9df40a4d20 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1100009598 0 0) 0x7f9dd8004ec0 con 0x7f9df4073d40 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9df4140690 con 0x7f9df4073d40 2024-09-16T03:05:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9de8001a80 con 0x7f9df4073d40 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): dump: 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184948115 0 0) 0x7f9df41403b0 con 0x7f9df40a4d20 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient(hunting): my global_id is 4321 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dd8004ec0 con 0x7f9df40a4d20 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2203413499 0 0) 0x7f9df4140690 con 0x7f9df4073d40 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 >> v1:172.21.15.46:6789/0 conn(0x7f9df40a4d20 legacy=0x7f9df41d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 >> v1:172.21.15.45:6789/0 conn(0x7f9df40a5810 legacy=0x7f9df41d5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient: _finish_hunting 0 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 monclient: found mon.c 2024-09-16T03:05:00.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df40a3c60 con 0x7f9df4073d40 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient: _finish_auth 0 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.958+0000 7f9df17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 5 monclient: authenticate success, global_id 4325 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 10 monclient: _renew_subs 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9df4140970 con 0x7f9df4073d40 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 10 monclient: _renew_subs 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9df413ee80 con 0x7f9df4073d40 2024-09-16T03:05:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 1 librados: init done 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df409d6d0 con 0x7f9df4073d40 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9de80038b0 con 0x7f9df4073d40 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9de8003cc0 con 0x7f9df4073d40 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 10 monclient: dump: 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df2ffd640 4 set_mon_vals no callback set 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f9de8003f20 con 0x7f9df4073d40 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f9de804b1d0 con 0x7f9df4073d40 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 10 client.4325.objecter ms_dispatch 0x7f9df4000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 3 client.4325.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:00.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 3 client.4325.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:00.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.962+0000 7f9df17fa640 20 client.4325.objecter dump_active .. 0 homeless 2024-09-16T03:05:00.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.966+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f9de8016bd0 con 0x7f9df4073d40 2024-09-16T03:05:00.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.966+0000 7f9df17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:00.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.966+0000 7f9df17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:00.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.966+0000 7f9df3fff640 10 monclient: get_auth_request con 0x7f9dd803e120 auth_method 0 2024-09-16T03:05:00.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.970+0000 7f9df17fa640 10 client.4325.objecter ms_handle_connect 0x7f9dd803e120 2024-09-16T03:05:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient: get_monmap_and_config 2024-09-16T03:05:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient: build_initial_monmap 2024-09-16T03:05:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient: monmap: 2024-09-16T03:05:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.992835+0000 2024-09-16T03:05:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.992835+0000 2024-09-16T03:05:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 1 Processor -- start 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 1 -- start start 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient: init 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient(hunting): picked mon.noname-a con 0x7fcd880a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient(hunting): picked mon.noname-b con 0x7fcd880a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient(hunting): picked mon.noname-c con 0x7fcd88073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd8809d9e0 con 0x7fcd880a4d20 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8d7c8640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fcd880a5810 0x7fcd8813cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35154/0 (socket says 172.21.15.45:35154) 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8d7c8640 1 -- 172.21.15.45:0/3241993506 learned_addr learned my addr 172.21.15.45:0/3241993506 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 1 -- 172.21.15.45:0/3241993506 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd8809dc20 con 0x7fcd88073d40 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 1 -- 172.21.15.45:0/3241993506 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd88073360 con 0x7fcd880a5810 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient(hunting): authenticate will time out at 991.642664s 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883528475 0 0) 0x7fcd8809dc20 con 0x7fcd88073d40 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient(hunting): _negotiate 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient(hunting): my global_id is 4324 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd74003530 con 0x7fcd88073d40 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 774614546 0 0) 0x7fcd88073360 con 0x7fcd880a5810 2024-09-16T03:05:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient(hunting): _negotiate 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient(hunting): my global_id is 4328 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd8809dc20 con 0x7fcd880a5810 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 66145467 0 0) 0x7fcd74003530 con 0x7fcd88073d40 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 >> v1:172.21.15.45:6790/0 conn(0x7fcd880a5810 legacy=0x7fcd8813cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 >> v1:172.21.15.45:6789/0 conn(0x7fcd880a4d20 legacy=0x7fcd880a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: _finish_hunting 0 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 monclient: found mon.noname-c 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8809cf30 con 0x7fcd88073d40 2024-09-16T03:05:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: _finish_auth 0 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: _check_auth_tickets 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd80001a20 con 0x7fcd88073d40 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: dump: 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 5 monclient: authenticate success, global_id 4324 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd80001c40 con 0x7fcd88073d40 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 1 -- 172.21.15.45:0/3241993506 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd80002090 con 0x7fcd88073d40 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd8cfc7640 10 monclient: dump: 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.002 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.990+0000 7fcd9022d640 4 set_mon_vals no callback set 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 10 monclient: shutdown 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 -- 172.21.15.45:0/3241993506 >> v1:172.21.15.46:6789/0 conn(0x7fcd88073d40 legacy=0x7fcd88140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 -- 172.21.15.45:0/3241993506 shutdown_connections 2024-09-16T03:05:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 -- 172.21.15.45:0/3241993506 >> 172.21.15.45:0/3241993506 conn(0x7fcd880aa720 msgr2=0x7fcd880aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 -- 172.21.15.45:0/3241993506 shutdown_connections 2024-09-16T03:05:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 -- 172.21.15.45:0/3241993506 wait complete. 2024-09-16T03:05:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 10 monclient: build_initial_monmap 2024-09-16T03:05:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 10 monclient: monmap: 2024-09-16T03:05:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 librados: starting msgr at 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 librados: starting objecter 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 Processor -- start 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 -- start start 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 librados: setting wanted keys 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 librados: calling monclient init 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 10 monclient: init 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 10 monclient(hunting): picked mon.noname-a con 0x7fcd88073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 10 monclient(hunting): picked mon.noname-b con 0x7fcd880a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 10 monclient(hunting): picked mon.noname-c con 0x7fcd880a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd8dfc9640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcd88073d40 0x7fcd8813ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42404/0 (socket says 172.21.15.45:42404) 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd8dfc9640 1 -- 172.21.15.45:0/2981822757 learned_addr learned my addr 172.21.15.45:0/2981822757 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd8e7ca640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fcd880a5810 0x7fcd881d51d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35160/0 (socket says 172.21.15.45:35160) 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd88074d90 con 0x7fcd88073d40 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd7e7fc640 10 client.?.objecter ms_handle_connect 0x7fcd880a4d20 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd7e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd7e7fc640 10 client.?.objecter ms_handle_connect 0x7fcd880a5810 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd7e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd7e7fc640 10 client.?.objecter ms_handle_connect 0x7fcd88073d40 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd7e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:01.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd88140270 con 0x7fcd880a4d20 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd8813d830 con 0x7fcd880a5810 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 monclient(hunting): authenticate will time out at 991.648077s 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1103926804 0 0) 0x7fcd88140270 con 0x7fcd880a4d20 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): my global_id is 4327 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd6c004ef0 con 0x7fcd880a4d20 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4209317159 0 0) 0x7fcd8809d9e0 con 0x7fcd88073d40 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:01.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): my global_id is 4305 2024-09-16T03:05:01.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd88140270 con 0x7fcd88073d40 2024-09-16T03:05:01.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1162978423 0 0) 0x7fcd8813d830 con 0x7fcd880a5810 2024-09-16T03:05:01.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:01.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:01.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:01.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): my global_id is 4331 2024-09-16T03:05:01.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd8809d9e0 con 0x7fcd880a5810 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fc88f657640 10 monclient: get_monmap_and_config 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fc88f657640 10 monclient: build_initial_monmap 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.994+0000 7fc88f657640 10 monclient: monmap: 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:00.999801+0000 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:00.999801+0000 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:01.012 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:01.012 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 1 Processor -- start 2024-09-16T03:05:01.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 1 -- start start 2024-09-16T03:05:01.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient: init 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1894911391 0 0) 0x7fcd6c004ef0 con 0x7fcd880a4d20 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd8813d830 con 0x7fcd880a4d20 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd78001940 con 0x7fcd880a4d20 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): dump: 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.013 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1316707974 0 0) 0x7fcd8809d9e0 con 0x7fcd880a5810 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd6c004ef0 con 0x7fcd880a5810 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd80001df0 con 0x7fcd880a5810 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): dump: 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.014 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1413728998 0 0) 0x7fcd88140270 con 0x7fcd88073d40 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd8809d9e0 con 0x7fcd88073d40 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd84001930 con 0x7fcd88073d40 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:01.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient(hunting): dump: 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2926527055 0 0) 0x7fcd8813d830 con 0x7fcd880a4d20 2024-09-16T03:05:01.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 >> v1:172.21.15.45:6790/0 conn(0x7fcd880a5810 legacy=0x7fcd881d51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 >> v1:172.21.15.45:6789/0 conn(0x7fcd88073d40 legacy=0x7fcd8813ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 monclient: found mon.b 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd88140550 con 0x7fcd880a4d20 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 5 monclient: authenticate success, global_id 4327 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 monclient: _renew_subs 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:01.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcd880a1e10 con 0x7fcd880a4d20 2024-09-16T03:05:01.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:01.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 monclient: _renew_subs 2024-09-16T03:05:01.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:01.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcd8813ff90 con 0x7fcd880a4d20 2024-09-16T03:05:01.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 1 librados: init done 2024-09-16T03:05:01.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:01.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:01.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient(hunting): picked mon.noname-c con 0x7fc88809a140 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:01.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient(hunting): picked mon.noname-b con 0x7fc88809ac30 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:01.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient(hunting): picked mon.noname-a con 0x7fc8881429f0 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:01.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8880a3a90 con 0x7fc8881429f0 2024-09-16T03:05:01.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88809bca0 con 0x7fc88809a140 2024-09-16T03:05:01.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88809b400 con 0x7fc88809ac30 2024-09-16T03:05:01.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:01.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient(hunting): authenticate will time out at 991.649291s 2024-09-16T03:05:01.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd780047f0 con 0x7fcd880a4d20 2024-09-16T03:05:01.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:01.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd78004bc0 con 0x7fcd880a4d20 2024-09-16T03:05:01.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:01.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 10 monclient: dump: 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7ffff640 4 set_mon_vals no callback set 2024-09-16T03:05:01.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:01.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:01.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:01.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88cbf2640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc88809ac30 0x7fc8881412d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35162/0 (socket says 172.21.15.45:35162) 2024-09-16T03:05:01.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88cbf2640 1 -- 172.21.15.45:0/2224065239 learned_addr learned my addr 172.21.15.45:0/2224065239 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:01.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1602099173 0 0) 0x7fc88809b400 con 0x7fc88809ac30 2024-09-16T03:05:01.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient(hunting): my global_id is 4334 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc874003530 con 0x7fc88809ac30 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2532570694 0 0) 0x7fc8880a3a90 con 0x7fc8881429f0 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient(hunting): my global_id is 4308 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc88809b400 con 0x7fc8881429f0 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2974744756 0 0) 0x7fc874003530 con 0x7fc88809ac30 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 >> v1:172.21.15.46:6789/0 conn(0x7fc88809a140 legacy=0x7fc88809a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 >> v1:172.21.15.45:6789/0 conn(0x7fc8881429f0 legacy=0x7fc888144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:05:01.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd70005090 con 0x7fcd880a4d20 2024-09-16T03:05:01.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 monclient: found mon.noname-b 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8880a2870 con 0x7fc88809ac30 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: _finish_auth 0 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fcd78004bc0 con 0x7fcd880a4d20 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fcd7804ada0 con 0x7fcd880a4d20 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fcd7e7fc640 10 client.4327.objecter ms_dispatch 0x7fcd88000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fcd7e7fc640 3 client.4327.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fcd7e7fc640 3 client.4327.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:01.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fcd7e7fc640 20 client.4327.objecter dump_active .. 0 homeless 2024-09-16T03:05:01.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:01.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc8780016a0 con 0x7fc88809ac30 2024-09-16T03:05:01.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:01.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:01.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: dump: 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fcd8dfc9640 10 monclient: get_auth_request con 0x7fcd6c03e3a0 auth_method 0 2024-09-16T03:05:01.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc878001840 con 0x7fc88809ac30 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 1 -- 172.21.15.45:0/2224065239 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc878001cd0 con 0x7fc88809ac30 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc87ffff640 10 monclient: dump: 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.030 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.031 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.031 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.031 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 5 monclient: authenticate success, global_id 4334 2024-09-16T03:05:01.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:01.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fcd7e7fc640 10 client.4327.objecter ms_handle_connect 0x7fcd6c03e3a0 2024-09-16T03:05:01.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 4 set_mon_vals no callback set 2024-09-16T03:05:01.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 10 monclient: shutdown 2024-09-16T03:05:01.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:01.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:00.998+0000 7fc88f657640 1 -- 172.21.15.45:0/2224065239 >> v1:172.21.15.45:6790/0 conn(0x7fc88809ac30 legacy=0x7fc8881412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 -- 172.21.15.45:0/2224065239 shutdown_connections 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 -- 172.21.15.45:0/2224065239 >> 172.21.15.45:0/2224065239 conn(0x7fc8880a2bd0 msgr2=0x7fc8880a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 -- 172.21.15.45:0/2224065239 shutdown_connections 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 -- 172.21.15.45:0/2224065239 wait complete. 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient: build_initial_monmap 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient: monmap: 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 librados: starting msgr at 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 librados: starting objecter 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 Processor -- start 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 -- start start 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 librados: setting wanted keys 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 librados: calling monclient init 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient: init 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient(hunting): picked mon.noname-b con 0x7fc88809a140 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient(hunting): picked mon.noname-a con 0x7fc88809ac30 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient(hunting): picked mon.noname-c con 0x7fc8881429f0 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88809d2c0 con 0x7fc88809ac30 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8880733a0 con 0x7fc88809a140 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc888073680 con 0x7fc8881429f0 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88f657640 10 monclient(hunting): authenticate will time out at 991.654585s 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88dbf4640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc8881429f0 0x7fc8881d9440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35174/0 (socket says 172.21.15.45:35174) 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88dbf4640 1 -- 172.21.15.45:0/2546292914 learned_addr learned my addr 172.21.15.45:0/2546292914 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc88cbf2640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc88809ac30 0x7fc888070010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42434/0 (socket says 172.21.15.45:42434) 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 client.?.objecter ms_handle_connect 0x7fc8881429f0 2024-09-16T03:05:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 client.?.objecter ms_handle_connect 0x7fc88809ac30 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 client.?.objecter ms_handle_connect 0x7fc88809a140 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668007503 0 0) 0x7fc8880a2870 con 0x7fc88809ac30 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): my global_id is 4311 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc86c004ec0 con 0x7fc88809ac30 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2899326893 0 0) 0x7fc888073680 con 0x7fc8881429f0 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): my global_id is 4337 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8880a2870 con 0x7fc8881429f0 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4284336204 0 0) 0x7fc86c004ec0 con 0x7fc88809ac30 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc888073680 con 0x7fc88809ac30 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc878001b90 con 0x7fc88809ac30 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): dump: 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.002+0000 7fc87dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1963681675 0 0) 0x7fc888073680 con 0x7fc88809ac30 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 >> v1:172.21.15.45:6790/0 conn(0x7fc8881429f0 legacy=0x7fc8881d9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 >> v1:172.21.15.46:6789/0 conn(0x7fc88809a140 legacy=0x7fc88806f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 1 monclient: found mon.a 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc888070720 con 0x7fc88809ac30 2024-09-16T03:05:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 monclient: _finish_auth 0 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 5 monclient: authenticate success, global_id 4311 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 10 monclient: _renew_subs 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc888074b60 con 0x7fc88809ac30 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 10 monclient: _renew_subs 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc888071e80 con 0x7fc88809ac30 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 1 librados: init done 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8780017a0 con 0x7fc88809ac30 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc878004da0 con 0x7fc88809ac30 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 monclient: dump: 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87f7fe640 4 set_mon_vals no callback set 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc88806c3e0 con 0x7fc88809ac30 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fc878010e10 con 0x7fc88809ac30 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fc87804afa0 con 0x7fc88809ac30 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 client.4311.objecter ms_dispatch 0x7fc888000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 3 client.4311.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 3 client.4311.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 20 client.4311.objecter dump_active .. 0 homeless 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc88d3f3640 10 monclient: get_auth_request con 0x7fc86c03dca0 auth_method 0 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.006+0000 7fc87dffb640 10 client.4311.objecter ms_handle_connect 0x7fc86c03dca0 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.014+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fc88809d2c0 con 0x7fc88809ac30 2024-09-16T03:05:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.014+0000 7fc87dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:01.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.014+0000 7fc87dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:01.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.026+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fcd78016820 con 0x7fcd880a4d20 2024-09-16T03:05:01.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.026+0000 7fcd7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:01.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.026+0000 7fcd7e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:01.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.142+0000 7fe4a410e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:05:01.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.142+0000 7fe4a410e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:05:01.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.142+0000 7fe4a410e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:01.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.142+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe49c09d6b0 con 0x7fe49c0a5810 2024-09-16T03:05:01.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.142+0000 7fe4927fc640 1 -- 172.21.15.45:0/2499095044 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 847495621) 0x7fe494016500 con 0x7fe49c0a5810 2024-09-16T03:05:01.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.142+0000 7fe4927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:05:01.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.142+0000 7fe4927fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:01.149 INFO:teuthology.orchestra.run.smithi045.stdout:47244640259 2024-09-16T03:05:01.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 10 librados: watch_flush enter 2024-09-16T03:05:01.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 10 librados: watch_flush exit 2024-09-16T03:05:01.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 20 client.4299.objecter shutdown clearing up homeless session... 2024-09-16T03:05:01.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 10 client.4299.objecter successfully canceled tick 2024-09-16T03:05:01.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 >> v1:172.21.15.46:6812/149360280 conn(0x7fe48003e4a0 legacy=0x7fe480040960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 10 monclient: shutdown 2024-09-16T03:05:01.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:01.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 >> v1:172.21.15.45:6789/0 conn(0x7fe49c0a5810 legacy=0x7fe49c1d5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 shutdown_connections 2024-09-16T03:05:01.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 >> 172.21.15.45:0/2499095044 conn(0x7fe49c0aa720 msgr2=0x7fe49c0a8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:01.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 shutdown_connections 2024-09-16T03:05:01.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 1 -- 172.21.15.45:0/2499095044 wait complete. 2024-09-16T03:05:01.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.146+0000 7fe4a410e640 1 librados: shutdown 2024-09-16T03:05:01.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2024-09-16T03:05:01.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.186+0000 7fc88f657640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:05:01.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.186+0000 7fc88f657640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:05:01.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.186+0000 7fc88f657640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:01.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.186+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc888076990 con 0x7fc88809ac30 2024-09-16T03:05:01.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.186+0000 7fc87dffb640 1 -- 172.21.15.45:0/2546292914 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 727498649) 0x7fc878016ac0 con 0x7fc88809ac30 2024-09-16T03:05:01.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.186+0000 7fc87dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:05:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.186+0000 7fc87dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:01.193 INFO:teuthology.orchestra.run.smithi045.stdout:51539607554 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 10 librados: watch_flush enter 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 10 librados: watch_flush exit 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 20 client.4311.objecter shutdown clearing up homeless session... 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 10 client.4311.objecter successfully canceled tick 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 >> v1:172.21.15.46:6812/149360280 conn(0x7fc86c03dca0 legacy=0x7fc86c040160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 10 monclient: shutdown 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 >> v1:172.21.15.45:6789/0 conn(0x7fc88809ac30 legacy=0x7fc888070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 shutdown_connections 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 >> 172.21.15.45:0/2546292914 conn(0x7fc8880a2bd0 msgr2=0x7fc888098800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 shutdown_connections 2024-09-16T03:05:01.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 1 -- 172.21.15.45:0/2546292914 wait complete. 2024-09-16T03:05:01.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.190+0000 7fc88f657640 1 librados: shutdown 2024-09-16T03:05:01.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607554 for osd.1 2024-09-16T03:05:01.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:05:01.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:05:01.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:01.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f249c1402f0 con 0x7f249c073d40 2024-09-16T03:05:01.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f2483fff640 1 -- 172.21.15.45:0/1568517313 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1444633209) 0x7f248c016a60 con 0x7f249c073d40 2024-09-16T03:05:01.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f2483fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:05:01.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f2483fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:01.218 INFO:teuthology.orchestra.run.smithi045.stdout:47244640260 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 10 librados: watch_flush enter 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 10 librados: watch_flush exit 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 20 client.4287.objecter shutdown clearing up homeless session... 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 10 client.4287.objecter successfully canceled tick 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 >> v1:172.21.15.46:6812/149360280 conn(0x7f248803e190 legacy=0x7f2488040650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 10 monclient: shutdown 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 >> v1:172.21.15.45:6789/0 conn(0x7f249c073d40 legacy=0x7f249c13c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 shutdown_connections 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 >> 172.21.15.45:0/1568517313 conn(0x7f249c0aa720 msgr2=0x7f249c0a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.214+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 shutdown_connections 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.218+0000 7f24a2138640 1 -- 172.21.15.45:0/1568517313 wait complete. 2024-09-16T03:05:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.218+0000 7f24a2138640 1 librados: shutdown 2024-09-16T03:05:01.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.222+0000 7f2d3e00d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:05:01.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:05:01.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:01.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f2d18005380 con 0x7f2d380a4d20 2024-09-16T03:05:01.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d1ffff640 1 -- 172.21.15.45:0/3105839754 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 956180462) 0x7f2d2c0164a0 con 0x7f2d380a4d20 2024-09-16T03:05:01.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:05:01.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d1ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:01.229 INFO:teuthology.orchestra.run.smithi045.stdout:51539607555 2024-09-16T03:05:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 10 librados: watch_flush enter 2024-09-16T03:05:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 10 librados: watch_flush exit 2024-09-16T03:05:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 20 client.4293.objecter shutdown clearing up homeless session... 2024-09-16T03:05:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 10 client.4293.objecter successfully canceled tick 2024-09-16T03:05:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 >> v1:172.21.15.46:6812/149360280 conn(0x7f2d24081f40 legacy=0x7f2d24084400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 10 monclient: shutdown 2024-09-16T03:05:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 >> v1:172.21.15.45:6789/0 conn(0x7f2d380a4d20 legacy=0x7f2d3809e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 shutdown_connections 2024-09-16T03:05:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 >> 172.21.15.45:0/3105839754 conn(0x7f2d380aa720 msgr2=0x7f2d381380b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 shutdown_connections 2024-09-16T03:05:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 1 -- 172.21.15.45:0/3105839754 wait complete. 2024-09-16T03:05:01.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.226+0000 7f2d3e00d640 1 librados: shutdown 2024-09-16T03:05:01.236 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.0 2024-09-16T03:05:01.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.5 2024-09-16T03:05:01.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:05:01.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:05:01.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:01.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9df4076b20 con 0x7f9df4073d40 2024-09-16T03:05:01.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9df17fa640 1 -- 172.21.15.45:0/3165632713 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 956180462) 0x7f9de80164d0 con 0x7f9df4073d40 2024-09-16T03:05:01.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9df17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:05:01.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9df17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:01.274 INFO:teuthology.orchestra.run.smithi045.stdout:51539607555 2024-09-16T03:05:01.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 10 librados: watch_flush enter 2024-09-16T03:05:01.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 10 librados: watch_flush exit 2024-09-16T03:05:01.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 20 client.4325.objecter shutdown clearing up homeless session... 2024-09-16T03:05:01.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 10 client.4325.objecter successfully canceled tick 2024-09-16T03:05:01.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 >> v1:172.21.15.46:6812/149360280 conn(0x7f9dd803e120 legacy=0x7f9dd80405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 10 monclient: shutdown 2024-09-16T03:05:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 >> v1:172.21.15.45:6790/0 conn(0x7f9df4073d40 legacy=0x7f9df41cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 shutdown_connections 2024-09-16T03:05:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 >> 172.21.15.45:0/3165632713 conn(0x7f9df40aa720 msgr2=0x7f9df40a4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 shutdown_connections 2024-09-16T03:05:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 1 -- 172.21.15.45:0/3165632713 wait complete. 2024-09-16T03:05:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.270+0000 7f9dfaf33640 1 librados: shutdown 2024-09-16T03:05:01.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.3 2024-09-16T03:05:01.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.310+0000 7fcd9022d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:05:01.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.310+0000 7fcd9022d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:05:01.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.310+0000 7fcd9022d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:05:01.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.310+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fcd70005380 con 0x7fcd880a4d20 2024-09-16T03:05:01.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd7e7fc640 1 -- 172.21.15.45:0/2981822757 <== mon.1 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 956180462) 0x7fcd78016120 con 0x7fcd880a4d20 2024-09-16T03:05:01.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:05:01.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd7e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:01.318 INFO:teuthology.orchestra.run.smithi045.stdout:51539607555 2024-09-16T03:05:01.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 10 librados: watch_flush enter 2024-09-16T03:05:01.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 10 librados: watch_flush exit 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 20 client.4327.objecter shutdown clearing up homeless session... 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 10 client.4327.objecter successfully canceled tick 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 >> v1:172.21.15.46:6812/149360280 conn(0x7fcd6c03e3a0 legacy=0x7fcd6c040860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 10 monclient: shutdown 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 >> v1:172.21.15.46:6789/0 conn(0x7fcd880a4d20 legacy=0x7fcd8813d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 shutdown_connections 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 >> 172.21.15.45:0/2981822757 conn(0x7fcd880aa720 msgr2=0x7fcd880a3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 shutdown_connections 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 1 -- 172.21.15.45:0/2981822757 wait complete. 2024-09-16T03:05:01.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:01.314+0000 7fcd9022d640 1 librados: shutdown 2024-09-16T03:05:01.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.4 2024-09-16T03:05:02.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:05:02.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:05:02.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:05:02.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:05:02.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:05:02.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient: get_monmap_and_config 2024-09-16T03:05:02.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient: monmap: 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:02.297157+0000 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:02.297157+0000 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:02.298 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 1 Processor -- start 2024-09-16T03:05:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 1 -- start start 2024-09-16T03:05:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient: init 2024-09-16T03:05:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient(hunting): picked mon.noname-a con 0x7f47340a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient(hunting): picked mon.noname-b con 0x7f47340a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient(hunting): picked mon.noname-c con 0x7f4734073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f473409d9e0 con 0x7f47340a4d20 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f473409dc20 con 0x7f4734073d40 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4734073360 con 0x7f47340a5810 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient(hunting): authenticate will time out at 992.946252s 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47337fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f47340a4d20 0x7f47340a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42442/0 (socket says 172.21.15.45:42442) 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47337fe640 1 -- 172.21.15.45:0/116470277 learned_addr learned my addr 172.21.15.45:0/116470277 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 418205800 0 0) 0x7f4734073360 con 0x7f47340a5810 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): my global_id is 4340 2024-09-16T03:05:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f471c003530 con 0x7f47340a5810 2024-09-16T03:05:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3185570215 0 0) 0x7f473409d9e0 con 0x7f47340a4d20 2024-09-16T03:05:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): my global_id is 4314 2024-09-16T03:05:02.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4734073360 con 0x7f47340a4d20 2024-09-16T03:05:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3048556337 0 0) 0x7f473409dc20 con 0x7f4734073d40 2024-09-16T03:05:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient(hunting): my global_id is 4333 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f473409d9e0 con 0x7f4734073d40 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2067835267 0 0) 0x7f4734073360 con 0x7f47340a4d20 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 >> v1:172.21.15.45:6790/0 conn(0x7f47340a5810 legacy=0x7f473413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 >> v1:172.21.15.46:6789/0 conn(0x7f4734073d40 legacy=0x7f4734140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 monclient: found mon.noname-a 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f473409cf30 con 0x7f47340a4d20 2024-09-16T03:05:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4724001600 con 0x7f47340a4d20 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 5 monclient: authenticate success, global_id 4314 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: dump: 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47240017a0 con 0x7f47340a4d20 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 1 -- 172.21.15.45:0/116470277 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4724001c30 con 0x7f47340a4d20 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f47327fc640 10 monclient: dump: 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.294+0000 7f4739ab9640 4 set_mon_vals no callback set 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: shutdown 2024-09-16T03:05:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/116470277 >> v1:172.21.15.45:6789/0 conn(0x7f47340a4d20 legacy=0x7f47340a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/116470277 shutdown_connections 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/116470277 >> 172.21.15.45:0/116470277 conn(0x7f47340aa720 msgr2=0x7f47340aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/116470277 shutdown_connections 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/116470277 wait complete. 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: monmap: 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 librados: starting msgr at 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 librados: starting objecter 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 Processor -- start 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- start start 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 librados: setting wanted keys 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 librados: calling monclient init 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: init 2024-09-16T03:05:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient(hunting): picked mon.noname-a con 0x7f4734073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient(hunting): picked mon.noname-c con 0x7f47340a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient(hunting): picked mon.noname-b con 0x7f47340a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f47337fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4734073d40 0x7f47341ccf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42454/0 (socket says 172.21.15.45:42454) 2024-09-16T03:05:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f47337fe640 1 -- 172.21.15.45:0/764204789 learned_addr learned my addr 172.21.15.45:0/764204789 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47341cdd90 con 0x7f4734073d40 2024-09-16T03:05:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47340a1510 con 0x7f47340a5810 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f473413f640 con 0x7f47340a4d20 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient(hunting): authenticate will time out at 992.950716s 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 client.?.objecter ms_handle_connect 0x7f47340a4d20 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 client.?.objecter ms_handle_connect 0x7f4734073d40 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 client.?.objecter ms_handle_connect 0x7f47340a5810 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4059194410 0 0) 0x7f473413f640 con 0x7f47340a4d20 2024-09-16T03:05:02.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): my global_id is 4343 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4720004ef0 con 0x7f47340a4d20 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92471255 0 0) 0x7f47341cdd90 con 0x7f4734073d40 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): my global_id is 4317 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f473413f640 con 0x7f4734073d40 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 413298503 0 0) 0x7f4720004ef0 con 0x7f47340a4d20 2024-09-16T03:05:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47341cdd90 con 0x7f47340a4d20 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4728001400 con 0x7f47340a4d20 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): dump: 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1027096749 0 0) 0x7f47340a1510 con 0x7f47340a5810 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): my global_id is 4336 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4720004ef0 con 0x7f47340a5810 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 791155435 0 0) 0x7f473413f640 con 0x7f4734073d40 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47340a1510 con 0x7f4734073d40 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4724001b70 con 0x7f4734073d40 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient(hunting): dump: 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 900407744 0 0) 0x7f47341cdd90 con 0x7f47340a4d20 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 >> v1:172.21.15.46:6789/0 conn(0x7f47340a5810 legacy=0x7f47341d51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 >> v1:172.21.15.45:6789/0 conn(0x7f4734073d40 legacy=0x7f47341ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 monclient: found mon.c 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f473413e150 con 0x7f47340a4d20 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f471bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 5 monclient: authenticate success, global_id 4343 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: _renew_subs 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47341404b0 con 0x7f47340a4d20 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: _renew_subs 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f473413ef10 con 0x7f47340a4d20 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.298+0000 7f4739ab9640 1 librados: init done 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47280015a0 con 0x7f47340a4d20 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4728003a50 con 0x7f47340a4d20 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 10 monclient: dump: 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.321 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f4731ffb640 4 set_mon_vals no callback set 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f4719ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f4719ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f4719ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f4719ffb640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c005090 con 0x7f47340a4d20 2024-09-16T03:05:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f472800fac0 con 0x7f47340a4d20 2024-09-16T03:05:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f4728049eb0 con 0x7f47340a4d20 2024-09-16T03:05:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 10 client.4343.objecter ms_dispatch 0x7f4734000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 3 client.4343.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 3 client.4343.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 20 client.4343.objecter dump_active .. 0 homeless 2024-09-16T03:05:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f47337fe640 10 monclient: get_auth_request con 0x7f472003e270 auth_method 0 2024-09-16T03:05:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.302+0000 7f471bfff640 10 client.4343.objecter ms_handle_connect 0x7f472003e270 2024-09-16T03:05:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.306+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4728015850 con 0x7f47340a4d20 2024-09-16T03:05:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.306+0000 7f471bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.306+0000 7f471bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:02.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:05:02.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.330+0000 7f7737e80640 10 monclient: get_monmap_and_config 2024-09-16T03:05:02.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.330+0000 7f7737e80640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.330+0000 7f7737e80640 10 monclient: monmap: 2024-09-16T03:05:02.336 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.336 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.336 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:02.335625+0000 2024-09-16T03:05:02.336 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:02.335625+0000 2024-09-16T03:05:02.337 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.337 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.337 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.337 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:02.337 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:02.337 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 Processor -- start 2024-09-16T03:05:02.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- start start 2024-09-16T03:05:02.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient: init 2024-09-16T03:05:02.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:02.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient(hunting): picked mon.noname-a con 0x7f77300a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient(hunting): picked mon.noname-b con 0x7f77300a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient(hunting): picked mon.noname-c con 0x7f7730073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773009d9e0 con 0x7f77300a4d20 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773009dc20 con 0x7f7730073d40 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7730073360 con 0x7f77300a5810 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient(hunting): authenticate will time out at 992.984722s 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f773541b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f77300a5810 0x7f773013cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35198/0 (socket says 172.21.15.45:35198) 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f773541b640 1 -- 172.21.15.45:0/2361803240 learned_addr learned my addr 172.21.15.45:0/2361803240 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 684150821 0 0) 0x7f7730073360 con 0x7f77300a5810 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient(hunting): my global_id is 4346 2024-09-16T03:05:02.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f771c003530 con 0x7f77300a5810 2024-09-16T03:05:02.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513943404 0 0) 0x7f773009d9e0 con 0x7f77300a4d20 2024-09-16T03:05:02.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient(hunting): my global_id is 4320 2024-09-16T03:05:02.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7730073360 con 0x7f77300a4d20 2024-09-16T03:05:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1888257642 0 0) 0x7f771c003530 con 0x7f77300a5810 2024-09-16T03:05:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 >> v1:172.21.15.46:6789/0 conn(0x7f7730073d40 legacy=0x7f7730140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 >> v1:172.21.15.45:6789/0 conn(0x7f77300a4d20 legacy=0x7f77300a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 monclient: found mon.noname-b 2024-09-16T03:05:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773009cf30 con 0x7f77300a5810 2024-09-16T03:05:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f77200016a0 con 0x7f77300a5810 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: dump: 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.347 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 5 monclient: authenticate success, global_id 4346 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77200018c0 con 0x7f77300a5810 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 1 -- 172.21.15.45:0/2361803240 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7720001d10 con 0x7f77300a5810 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 4 set_mon_vals no callback set 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7734c1a640 10 monclient: dump: 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient: shutdown 2024-09-16T03:05:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- 172.21.15.45:0/2361803240 >> v1:172.21.15.45:6790/0 conn(0x7f77300a5810 legacy=0x7f773013cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- 172.21.15.45:0/2361803240 shutdown_connections 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- 172.21.15.45:0/2361803240 >> 172.21.15.45:0/2361803240 conn(0x7f77300aa720 msgr2=0x7f77300aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- 172.21.15.45:0/2361803240 shutdown_connections 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- 172.21.15.45:0/2361803240 wait complete. 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient: monmap: 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 librados: starting msgr at 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 librados: starting objecter 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 Processor -- start 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 -- start start 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 librados: setting wanted keys 2024-09-16T03:05:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 1 librados: calling monclient init 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.334+0000 7f7737e80640 10 monclient: init 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient(hunting): picked mon.noname-b con 0x7f7730073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient(hunting): picked mon.noname-a con 0x7f77300a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient(hunting): picked mon.noname-c con 0x7f77300a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773013c490 con 0x7f77300a4d20 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7730139d30 con 0x7f7730073d40 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773013a130 con 0x7f77300a5810 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient(hunting): authenticate will time out at 992.988176s 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f773541b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f77300a4d20 0x7f7730139620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42470/0 (socket says 172.21.15.45:42470) 2024-09-16T03:05:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f773541b640 1 -- 172.21.15.45:0/1166081224 learned_addr learned my addr 172.21.15.45:0/1166081224 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 client.?.objecter ms_handle_connect 0x7f77300a5810 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 client.?.objecter ms_handle_connect 0x7f77300a4d20 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 client.?.objecter ms_handle_connect 0x7f7730073d40 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164616344 0 0) 0x7f773013a130 con 0x7f77300a5810 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): my global_id is 4349 2024-09-16T03:05:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7714004ef0 con 0x7f77300a5810 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1966691319 0 0) 0x7f773013c490 con 0x7f77300a4d20 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): my global_id is 4323 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f773013a130 con 0x7f77300a4d20 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298790501 0 0) 0x7f7730139d30 con 0x7f7730073d40 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): my global_id is 4342 2024-09-16T03:05:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f773013c490 con 0x7f7730073d40 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3727824075 0 0) 0x7f7714004ef0 con 0x7f77300a5810 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7730139d30 con 0x7f77300a5810 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7728001bd0 con 0x7f77300a5810 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): dump: 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 935333490 0 0) 0x7f773013a130 con 0x7f77300a4d20 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7714004ef0 con 0x7f77300a4d20 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7720001c10 con 0x7f77300a4d20 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): dump: 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2753711475 0 0) 0x7f773013c490 con 0x7f7730073d40 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f773013a130 con 0x7f7730073d40 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f772c001940 con 0x7f7730073d40 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient(hunting): dump: 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 816493012 0 0) 0x7f7730139d30 con 0x7f77300a5810 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 >> v1:172.21.15.46:6789/0 conn(0x7f7730073d40 legacy=0x7f7730138f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 >> v1:172.21.15.45:6789/0 conn(0x7f77300a4d20 legacy=0x7f7730139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 monclient: found mon.c 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77300a1e40 con 0x7f77300a5810 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 5 monclient: authenticate success, global_id 4349 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient: _renew_subs 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f773013b620 con 0x7f77300a5810 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient: _renew_subs 2024-09-16T03:05:02.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f773013aef0 con 0x7f77300a5810 2024-09-16T03:05:02.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 1 librados: init done 2024-09-16T03:05:02.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f773009d6d0 con 0x7f77300a5810 2024-09-16T03:05:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7728003980 con 0x7f77300a5810 2024-09-16T03:05:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7728003cd0 con 0x7f77300a5810 2024-09-16T03:05:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f77267fc640 10 monclient: dump: 2024-09-16T03:05:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.338+0000 7f7727fff640 4 set_mon_vals no callback set 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.342+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f772800fd40 con 0x7f77300a5810 2024-09-16T03:05:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.342+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f7728049f40 con 0x7f77300a5810 2024-09-16T03:05:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.342+0000 7f77267fc640 10 client.4349.objecter ms_dispatch 0x7f7730000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.342+0000 7f77267fc640 3 client.4349.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.342+0000 7f77267fc640 3 client.4349.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.342+0000 7f77267fc640 20 client.4349.objecter dump_active .. 0 homeless 2024-09-16T03:05:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.342+0000 7f7735c1c640 10 monclient: get_auth_request con 0x7f771403e3d0 auth_method 0 2024-09-16T03:05:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.342+0000 7f77267fc640 10 client.4349.objecter ms_handle_connect 0x7f771403e3d0 2024-09-16T03:05:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.346+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f7728078d40 con 0x7f77300a5810 2024-09-16T03:05:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.346+0000 7f77267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.346+0000 7f77267fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient: get_monmap_and_config 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient: monmap: 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:02.376379+0000 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:02.376379+0000 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:02.377 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:02.378 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 Processor -- start 2024-09-16T03:05:02.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 -- start start 2024-09-16T03:05:02.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient: init 2024-09-16T03:05:02.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:05:02.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient(hunting): picked mon.noname-b con 0x7f0a600a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient(hunting): picked mon.noname-a con 0x7f0a600a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient(hunting): picked mon.noname-c con 0x7f0a60073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a6009d9e0 con 0x7f0a600a5810 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a6009dc20 con 0x7f0a60073d40 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a60073360 con 0x7f0a600a4d20 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient(hunting): authenticate will time out at 993.025559s 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5effd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0a600a4d20 0x7f0a600a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35220/0 (socket says 172.21.15.45:35220) 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5effd640 1 -- 172.21.15.45:0/2333924328 learned_addr learned my addr 172.21.15.45:0/2333924328 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 643545583 0 0) 0x7f0a60073360 con 0x7f0a600a4d20 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient(hunting): my global_id is 4352 2024-09-16T03:05:02.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a4c003530 con 0x7f0a600a4d20 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2608440243 0 0) 0x7f0a6009d9e0 con 0x7f0a600a5810 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient(hunting): my global_id is 4326 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a60073360 con 0x7f0a600a5810 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1903701184 0 0) 0x7f0a4c003530 con 0x7f0a600a4d20 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 >> v1:172.21.15.46:6789/0 conn(0x7f0a60073d40 legacy=0x7f0a60140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 >> v1:172.21.15.45:6789/0 conn(0x7f0a600a5810 legacy=0x7f0a6013cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 monclient: found mon.noname-b 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6009cf30 con 0x7f0a600a4d20 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 5 monclient: authenticate success, global_id 4352 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0a500016a0 con 0x7f0a600a4d20 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: dump: 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a50001840 con 0x7f0a600a4d20 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 1 -- 172.21.15.45:0/2333924328 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0a50001c90 con 0x7f0a600a4d20 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 4 set_mon_vals no callback set 2024-09-16T03:05:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a5e7fc640 10 monclient: dump: 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.384 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 10 monclient: shutdown 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 -- 172.21.15.45:0/2333924328 >> v1:172.21.15.45:6790/0 conn(0x7f0a600a4d20 legacy=0x7f0a600a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 -- 172.21.15.45:0/2333924328 shutdown_connections 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 -- 172.21.15.45:0/2333924328 >> 172.21.15.45:0/2333924328 conn(0x7f0a600aa720 msgr2=0x7f0a600aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 -- 172.21.15.45:0/2333924328 shutdown_connections 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.374+0000 7f0a65983640 1 -- 172.21.15.45:0/2333924328 wait complete. 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient: monmap: 2024-09-16T03:05:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 librados: starting msgr at 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 librados: starting objecter 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 Processor -- start 2024-09-16T03:05:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 -- start start 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 librados: setting wanted keys 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 librados: calling monclient init 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient: init 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient(hunting): picked mon.noname-c con 0x7f0a60073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient(hunting): picked mon.noname-b con 0x7f0a600a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient(hunting): picked mon.noname-a con 0x7f0a600a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a60123cf0 con 0x7f0a600a5810 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a600a1510 con 0x7f0a600a4d20 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a60122230 con 0x7f0a60073d40 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient(hunting): authenticate will time out at 993.029402s 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0a600a5810 0x7f0a601d99b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42490/0 (socket says 172.21.15.45:42490) 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5f7fe640 1 -- 172.21.15.45:0/2703235033 learned_addr learned my addr 172.21.15.45:0/2703235033 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 client.?.objecter ms_handle_connect 0x7f0a600a5810 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 client.?.objecter ms_handle_connect 0x7f0a60073d40 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495548380 0 0) 0x7f0a60122230 con 0x7f0a60073d40 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): my global_id is 4355 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a40004ec0 con 0x7f0a60073d40 2024-09-16T03:05:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3215267377 0 0) 0x7f0a6009dc20 con 0x7f0a600a5810 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): my global_id is 4329 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a60122230 con 0x7f0a600a5810 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 415401052 0 0) 0x7f0a40004ec0 con 0x7f0a60073d40 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a6009dc20 con 0x7f0a60073d40 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 client.?.objecter ms_handle_connect 0x7f0a600a4d20 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0a50001b90 con 0x7f0a60073d40 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): dump: 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1611795100 0 0) 0x7f0a60122230 con 0x7f0a600a5810 2024-09-16T03:05:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a40004ec0 con 0x7f0a600a5810 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0a58001d00 con 0x7f0a600a5810 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient(hunting): dump: 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1111235434 0 0) 0x7f0a6009dc20 con 0x7f0a60073d40 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 >> v1:172.21.15.46:6789/0 conn(0x7f0a600a4d20 legacy=0x7f0a601d6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 >> v1:172.21.15.45:6789/0 conn(0x7f0a600a5810 legacy=0x7f0a601d99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 monclient: found mon.c 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a60123fd0 con 0x7f0a60073d40 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 5 monclient: authenticate success, global_id 4355 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient: _renew_subs 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0a601389f0 con 0x7f0a60073d40 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient: _renew_subs 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0a60122d90 con 0x7f0a60073d40 2024-09-16T03:05:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 1 librados: init done 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a50001d30 con 0x7f0a60073d40 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0a50004da0 con 0x7f0a60073d40 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5cff9640 10 monclient: dump: 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a5dffb640 4 set_mon_vals no callback set 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.378+0000 7f0a65983640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.382+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f0a50010e10 con 0x7f0a60073d40 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.382+0000 7f0a65983640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.382+0000 7f0a65983640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.382+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a6009d6b0 con 0x7f0a60073d40 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.382+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f0a5004af90 con 0x7f0a60073d40 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.382+0000 7f0a5cff9640 10 client.4355.objecter ms_dispatch 0x7f0a60000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.382+0000 7f0a5cff9640 3 client.4355.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.382+0000 7f0a5cff9640 3 client.4355.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.382+0000 7f0a5cff9640 20 client.4355.objecter dump_active .. 0 homeless 2024-09-16T03:05:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.386+0000 7f0a57fff640 10 monclient: get_auth_request con 0x7f0a4003e0d0 auth_method 0 2024-09-16T03:05:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.386+0000 7f0a5cff9640 10 client.4355.objecter ms_handle_connect 0x7f0a4003e0d0 2024-09-16T03:05:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.390+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f0a50016a50 con 0x7f0a60073d40 2024-09-16T03:05:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.390+0000 7f0a5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.390+0000 7f0a5cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:02.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient: get_monmap_and_config 2024-09-16T03:05:02.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient: monmap: 2024-09-16T03:05:02.413 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.413 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.414 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:02.412754+0000 2024-09-16T03:05:02.414 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:02.412754+0000 2024-09-16T03:05:02.414 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.414 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.414 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.414 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:02.414 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:02.414 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 1 Processor -- start 2024-09-16T03:05:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 1 -- start start 2024-09-16T03:05:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient: init 2024-09-16T03:05:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient(hunting): picked mon.noname-a con 0x7f86640a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient(hunting): picked mon.noname-c con 0x7f86640a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient(hunting): picked mon.noname-b con 0x7f8664073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f866409d9e0 con 0x7f86640a4d20 2024-09-16T03:05:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f866409dc20 con 0x7f86640a5810 2024-09-16T03:05:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8664073360 con 0x7f8664073d40 2024-09-16T03:05:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient(hunting): authenticate will time out at 993.062371s 2024-09-16T03:05:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f8663fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f86640a4d20 0x7f86640a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42498/0 (socket says 172.21.15.45:42498) 2024-09-16T03:05:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f8663fff640 1 -- 172.21.15.45:0/183412968 learned_addr learned my addr 172.21.15.45:0/183412968 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566911583 0 0) 0x7f8664073360 con 0x7f8664073d40 2024-09-16T03:05:02.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): my global_id is 4358 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8650003530 con 0x7f8664073d40 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182169116 0 0) 0x7f866409d9e0 con 0x7f86640a4d20 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): my global_id is 4332 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86500048f0 con 0x7f86640a4d20 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249879617 0 0) 0x7f866409dc20 con 0x7f86640a5810 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient(hunting): my global_id is 4351 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f866409d9e0 con 0x7f86640a5810 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 668637816 0 0) 0x7f86500048f0 con 0x7f86640a4d20 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 >> v1:172.21.15.45:6790/0 conn(0x7f8664073d40 legacy=0x7f8664140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 >> v1:172.21.15.46:6789/0 conn(0x7f86640a5810 legacy=0x7f866413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 monclient: found mon.noname-a 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866409cf30 con 0x7f86640a4d20 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8654001600 con 0x7f86640a4d20 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: dump: 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 5 monclient: authenticate success, global_id 4332 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86540017a0 con 0x7f86640a4d20 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 1 -- 172.21.15.45:0/183412968 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8654001c30 con 0x7f86640a4d20 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f86637fe640 10 monclient: dump: 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.410+0000 7f866a67c640 4 set_mon_vals no callback set 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.414+0000 7f866a67c640 10 monclient: shutdown 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.414+0000 7f866a67c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.414+0000 7f866a67c640 1 -- 172.21.15.45:0/183412968 >> v1:172.21.15.45:6789/0 conn(0x7f86640a4d20 legacy=0x7f86640a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.414+0000 7f866a67c640 1 -- 172.21.15.45:0/183412968 shutdown_connections 2024-09-16T03:05:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.414+0000 7f866a67c640 1 -- 172.21.15.45:0/183412968 >> 172.21.15.45:0/183412968 conn(0x7f86640aa720 msgr2=0x7f86640aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 -- 172.21.15.45:0/183412968 shutdown_connections 2024-09-16T03:05:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 -- 172.21.15.45:0/183412968 wait complete. 2024-09-16T03:05:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient: monmap: 2024-09-16T03:05:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 librados: starting msgr at 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 librados: starting objecter 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 Processor -- start 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 -- start start 2024-09-16T03:05:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 librados: setting wanted keys 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 librados: calling monclient init 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient: init 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient(hunting): picked mon.noname-a con 0x7f8664073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient(hunting): picked mon.noname-b con 0x7f86640a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient(hunting): picked mon.noname-c con 0x7f86640a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86641224f0 con 0x7f8664073d40 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8664124850 con 0x7f86640a4d20 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8664123fb0 con 0x7f86640a5810 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient(hunting): authenticate will time out at 993.070410s 2024-09-16T03:05:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f8663fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8664073d40 0x7f8664137ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42514/0 (socket says 172.21.15.45:42514) 2024-09-16T03:05:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f8663fff640 1 -- 172.21.15.45:0/5069588 learned_addr learned my addr 172.21.15.45:0/5069588 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 client.?.objecter ms_handle_connect 0x7f8664073d40 2024-09-16T03:05:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 client.?.objecter ms_handle_connect 0x7f86640a5810 2024-09-16T03:05:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 client.?.objecter ms_handle_connect 0x7f86640a4d20 2024-09-16T03:05:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1908420548 0 0) 0x7f86641224f0 con 0x7f8664073d40 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): my global_id is 4335 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8644004ef0 con 0x7f8664073d40 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979679546 0 0) 0x7f8664123fb0 con 0x7f86640a5810 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): my global_id is 4361 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86641224f0 con 0x7f86640a5810 2024-09-16T03:05:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213981801 0 0) 0x7f8664124850 con 0x7f86640a4d20 2024-09-16T03:05:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): my global_id is 4354 2024-09-16T03:05:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8664123fb0 con 0x7f86640a4d20 2024-09-16T03:05:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1724486112 0 0) 0x7f8644004ef0 con 0x7f8664073d40 2024-09-16T03:05:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8664124850 con 0x7f8664073d40 2024-09-16T03:05:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8654001b30 con 0x7f8664073d40 2024-09-16T03:05:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): dump: 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.432 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2644899254 0 0) 0x7f86641224f0 con 0x7f86640a5810 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8644004ef0 con 0x7f86640a5810 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f865c001bb0 con 0x7f86640a5810 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient(hunting): dump: 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.434 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 234572376 0 0) 0x7f8664124850 con 0x7f8664073d40 2024-09-16T03:05:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 >> v1:172.21.15.45:6790/0 conn(0x7f86640a5810 legacy=0x7f86641d9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 >> v1:172.21.15.46:6789/0 conn(0x7f86640a4d20 legacy=0x7f86641d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 monclient: found mon.a 2024-09-16T03:05:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86641385b0 con 0x7f8664073d40 2024-09-16T03:05:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f86617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 5 monclient: authenticate success, global_id 4335 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient: _renew_subs 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8664123050 con 0x7f8664073d40 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient: _renew_subs 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.418+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8664120f10 con 0x7f8664073d40 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f866a67c640 1 librados: init done 2024-09-16T03:05:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8654004c00 con 0x7f8664073d40 2024-09-16T03:05:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8654004db0 con 0x7f8664073d40 2024-09-16T03:05:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f8662ffd640 4 set_mon_vals no callback set 2024-09-16T03:05:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 10 monclient: dump: 2024-09-16T03:05:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f866a67c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f8654010e20 con 0x7f8664073d40 2024-09-16T03:05:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f866a67c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f866a67c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8648005090 con 0x7f8664073d40 2024-09-16T03:05:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f865404afe0 con 0x7f8664073d40 2024-09-16T03:05:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 10 client.4335.objecter ms_dispatch 0x7f8664000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 3 client.4335.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 3 client.4335.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 20 client.4335.objecter dump_active .. 0 homeless 2024-09-16T03:05:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f865b7fe640 10 monclient: get_auth_request con 0x7f864403e320 auth_method 0 2024-09-16T03:05:02.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.422+0000 7f86617fa640 10 client.4335.objecter ms_handle_connect 0x7f864403e320 2024-09-16T03:05:02.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.426+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f8654016aa0 con 0x7f8664073d40 2024-09-16T03:05:02.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.430+0000 7f86617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.430+0000 7f86617fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:02.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient: get_monmap_and_config 2024-09-16T03:05:02.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient: monmap: 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:02.456945+0000 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:02.456945+0000 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:02.458 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 1 Processor -- start 2024-09-16T03:05:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 1 -- start start 2024-09-16T03:05:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient: init 2024-09-16T03:05:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient(hunting): picked mon.noname-c con 0x7fb8c00a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient(hunting): picked mon.noname-a con 0x7fb8c00a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bf7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb8c0073d40 0x7fb8c0140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35246/0 (socket says 172.21.15.45:35246) 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8beffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb8c00a4d20 0x7fb8c00a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.45:39770/0 (socket says 172.21.15.45:39770) 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8be7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb8c00a5810 0x7fb8c013cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42526/0 (socket says 172.21.15.45:42526) 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bf7fe640 1 -- 172.21.15.45:0/998175422 learned_addr learned my addr 172.21.15.45:0/998175422 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient(hunting): picked mon.noname-b con 0x7fb8c0073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 1 -- 172.21.15.45:0/998175422 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c009d9e0 con 0x7fb8c00a5810 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 1 -- 172.21.15.45:0/998175422 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c009dc20 con 0x7fb8c00a4d20 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 1 -- 172.21.15.45:0/998175422 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c0073360 con 0x7fb8c0073d40 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3179175191 0 0) 0x7fb8c009d9e0 con 0x7fb8c00a5810 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8c5697640 10 monclient(hunting): authenticate will time out at 993.107515s 2024-09-16T03:05:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): my global_id is 4338 2024-09-16T03:05:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8a8003530 con 0x7fb8c00a5810 2024-09-16T03:05:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2365470209 0 0) 0x7fb8c009dc20 con 0x7fb8c00a4d20 2024-09-16T03:05:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): my global_id is 4357 2024-09-16T03:05:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8c009d9e0 con 0x7fb8c00a4d20 2024-09-16T03:05:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266839694 0 0) 0x7fb8c0073360 con 0x7fb8c0073d40 2024-09-16T03:05:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 10 monclient(hunting): my global_id is 4364 2024-09-16T03:05:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8c009dc20 con 0x7fb8c0073d40 2024-09-16T03:05:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1849772767 0 0) 0x7fb8c009dc20 con 0x7fb8c0073d40 2024-09-16T03:05:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.454+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 >> v1:172.21.15.46:6789/0 conn(0x7fb8c00a4d20 legacy=0x7fb8c00a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 >> v1:172.21.15.45:6789/0 conn(0x7fb8c00a5810 legacy=0x7fb8c013cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 1 monclient: found mon.noname-b 2024-09-16T03:05:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c009cf30 con 0x7fb8c0073d40 2024-09-16T03:05:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb8b80018c0 con 0x7fb8c0073d40 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: dump: 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8b8001a60 con 0x7fb8c0073d40 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 1 -- 172.21.15.45:0/998175422 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb8b8001eb0 con 0x7fb8c0073d40 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8bdffb640 10 monclient: dump: 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 5 monclient: authenticate success, global_id 4364 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 4 set_mon_vals no callback set 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 10 monclient: shutdown 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 -- 172.21.15.45:0/998175422 >> v1:172.21.15.45:6790/0 conn(0x7fb8c0073d40 legacy=0x7fb8c0140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 -- 172.21.15.45:0/998175422 shutdown_connections 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 -- 172.21.15.45:0/998175422 >> 172.21.15.45:0/998175422 conn(0x7fb8c00aa720 msgr2=0x7fb8c00aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 -- 172.21.15.45:0/998175422 shutdown_connections 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 -- 172.21.15.45:0/998175422 wait complete. 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 10 monclient: monmap: 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 librados: starting msgr at 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 librados: starting objecter 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 Processor -- start 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 -- start start 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 librados: setting wanted keys 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 1 librados: calling monclient init 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 10 monclient: init 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 10 monclient(hunting): picked mon.noname-c con 0x7fb8c0073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.458+0000 7fb8c5697640 10 monclient(hunting): picked mon.noname-b con 0x7fb8c00a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 monclient(hunting): picked mon.noname-a con 0x7fb8c00a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8beffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb8c0073d40 0x7fb8c013c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35254/0 (socket says 172.21.15.45:35254) 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8beffd640 1 -- 172.21.15.45:0/1651412244 learned_addr learned my addr 172.21.15.45:0/1651412244 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 client.?.objecter ms_handle_connect 0x7fb8c0073d40 2024-09-16T03:05:02.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 client.?.objecter ms_handle_connect 0x7fb8c00a5810 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c0140750 con 0x7fb8c00a5810 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 client.?.objecter ms_handle_connect 0x7fb8c00a4d20 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c013d750 con 0x7fb8c00a4d20 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c0074d90 con 0x7fb8c0073d40 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 monclient(hunting): authenticate will time out at 993.112721s 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1439824783 0 0) 0x7fb8c0140750 con 0x7fb8c00a5810 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): my global_id is 4341 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8ac004ec0 con 0x7fb8c00a5810 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 636840270 0 0) 0x7fb8c013d750 con 0x7fb8c00a4d20 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): my global_id is 4360 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8c0140750 con 0x7fb8c00a4d20 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 14072583 0 0) 0x7fb8ac004ec0 con 0x7fb8c00a5810 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8c013d750 con 0x7fb8c00a5810 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 553573547 0 0) 0x7fb8c0074d90 con 0x7fb8c0073d40 2024-09-16T03:05:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): my global_id is 4367 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8ac004ec0 con 0x7fb8c0073d40 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb8b8001db0 con 0x7fb8c00a5810 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): dump: 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.477 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2697545197 0 0) 0x7fb8c0140750 con 0x7fb8c00a4d20 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8c0074d90 con 0x7fb8c00a4d20 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3215867937 0 0) 0x7fb8c013d750 con 0x7fb8c00a5810 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 >> v1:172.21.15.45:6790/0 conn(0x7fb8c0073d40 legacy=0x7fb8c013c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 >> v1:172.21.15.46:6789/0 conn(0x7fb8c00a4d20 legacy=0x7fb8c013d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 monclient: found mon.a 2024-09-16T03:05:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c013f8e0 con 0x7fb8c00a5810 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 5 monclient: authenticate success, global_id 4341 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 monclient: _renew_subs 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c013e3f0 con 0x7fb8c00a5810 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 monclient: _renew_subs 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb8c01389e0 con 0x7fb8c00a5810 2024-09-16T03:05:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 1 librados: init done 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8b8004e20 con 0x7fb8c00a5810 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb8b8005170 con 0x7fb8c00a5810 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8bd7fa640 4 set_mon_vals no callback set 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8a77fe640 10 monclient: dump: 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.481 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.481 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.462+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8a0005090 con 0x7fb8c00a5810 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.466+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fb8b8011040 con 0x7fb8c00a5810 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.466+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fb8b804af70 con 0x7fb8c00a5810 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.466+0000 7fb8a77fe640 10 client.4341.objecter ms_dispatch 0x7fb8c0000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.466+0000 7fb8a77fe640 3 client.4341.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.466+0000 7fb8a77fe640 3 client.4341.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.466+0000 7fb8a77fe640 20 client.4341.objecter dump_active .. 0 homeless 2024-09-16T03:05:02.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.466+0000 7fb8beffd640 10 monclient: get_auth_request con 0x7fb8ac03e0a0 auth_method 0 2024-09-16T03:05:02.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.466+0000 7fb8a77fe640 10 client.4341.objecter ms_handle_connect 0x7fb8ac03e0a0 2024-09-16T03:05:02.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.474+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fb8b8016a30 con 0x7fb8c00a5810 2024-09-16T03:05:02.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.474+0000 7fb8a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.474+0000 7fb8a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:02.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient: get_monmap_and_config 2024-09-16T03:05:02.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient: monmap: 2024-09-16T03:05:02.490 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.490 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.490 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:02.489905+0000 2024-09-16T03:05:02.491 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:02.489905+0000 2024-09-16T03:05:02.491 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.491 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.491 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.491 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:02.491 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:02.491 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 1 Processor -- start 2024-09-16T03:05:02.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 1 -- start start 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient: init 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient(hunting): picked mon.noname-c con 0x7f1f4c0a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient(hunting): picked mon.noname-a con 0x7f1f4c0a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f52bf3640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1f4c073d40 0x7f1f4c140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35260/0 (socket says 172.21.15.45:35260) 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f51bf1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1f4c0a5810 0x7f1f4c13cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42556/0 (socket says 172.21.15.45:42556) 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f52bf3640 1 -- 172.21.15.45:0/1562160331 learned_addr learned my addr 172.21.15.45:0/1562160331 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient(hunting): picked mon.noname-b con 0x7f1f4c073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 1 -- 172.21.15.45:0/1562160331 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f4c09d9e0 con 0x7f1f4c0a5810 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 1 -- 172.21.15.45:0/1562160331 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f4c09dc20 con 0x7f1f4c0a4d20 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632865029 0 0) 0x7f1f4c09d9e0 con 0x7f1f4c0a5810 2024-09-16T03:05:02.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 1 -- 172.21.15.45:0/1562160331 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f4c073360 con 0x7f1f4c073d40 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.486+0000 7f1f54656640 10 monclient(hunting): authenticate will time out at 993.139926s 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient(hunting): my global_id is 4344 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f38003530 con 0x7f1f4c0a5810 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2102846586 0 0) 0x7f1f4c073360 con 0x7f1f4c073d40 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient(hunting): my global_id is 4370 2024-09-16T03:05:02.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f4c09d9e0 con 0x7f1f4c073d40 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2383522745 0 0) 0x7f1f38003530 con 0x7f1f4c0a5810 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 >> v1:172.21.15.45:6790/0 conn(0x7f1f4c073d40 legacy=0x7f1f4c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 >> v1:172.21.15.46:6789/0 conn(0x7f1f4c0a4d20 legacy=0x7f1f4c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 monclient: found mon.noname-a 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4c09cf30 con 0x7f1f4c0a5810 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1f3c0016a0 con 0x7f1f4c0a5810 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: dump: 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f54656640 5 monclient: authenticate success, global_id 4344 2024-09-16T03:05:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f54656640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1f3c001840 con 0x7f1f4c0a5810 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 1 -- 172.21.15.45:0/1562160331 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1f3c001cd0 con 0x7f1f4c0a5810 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f513f0640 10 monclient: dump: 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.501 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f54656640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f54656640 4 set_mon_vals no callback set 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f54656640 10 monclient: shutdown 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f54656640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f54656640 1 -- 172.21.15.45:0/1562160331 >> v1:172.21.15.45:6789/0 conn(0x7f1f4c0a5810 legacy=0x7f1f4c13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f54656640 1 -- 172.21.15.45:0/1562160331 shutdown_connections 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.490+0000 7f1f54656640 1 -- 172.21.15.45:0/1562160331 >> 172.21.15.45:0/1562160331 conn(0x7f1f4c0aa720 msgr2=0x7f1f4c0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 -- 172.21.15.45:0/1562160331 shutdown_connections 2024-09-16T03:05:02.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 -- 172.21.15.45:0/1562160331 wait complete. 2024-09-16T03:05:02.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient: monmap: 2024-09-16T03:05:02.506 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.506 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 librados: starting msgr at 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 librados: starting objecter 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 Processor -- start 2024-09-16T03:05:02.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 -- start start 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 librados: setting wanted keys 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 librados: calling monclient init 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient: init 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient(hunting): picked mon.noname-b con 0x7f1f4c073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient(hunting): picked mon.noname-c con 0x7f1f4c0a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient(hunting): picked mon.noname-a con 0x7f1f4c1d31c0 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f51bf1640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1f4c0a5810 0x7f1f4c13d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35268/0 (socket says 172.21.15.45:35268) 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f51bf1640 1 -- 172.21.15.45:0/1389594244 learned_addr learned my addr 172.21.15.45:0/1389594244 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f4c0a1510 con 0x7f1f4c1d31c0 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f4c13d9e0 con 0x7f1f4c073d40 2024-09-16T03:05:02.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f4c140920 con 0x7f1f4c0a5810 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 client.?.objecter ms_handle_connect 0x7f1f4c0a5810 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient(hunting): authenticate will time out at 993.146559s 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174580626 0 0) 0x7f1f4c140920 con 0x7f1f4c0a5810 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): my global_id is 4373 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f30004ec0 con 0x7f1f4c0a5810 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 client.?.objecter ms_handle_connect 0x7f1f4c1d31c0 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2473375243 0 0) 0x7f1f30004ec0 con 0x7f1f4c0a5810 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f4c140920 con 0x7f1f4c0a5810 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1f3c001c50 con 0x7f1f4c0a5810 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): dump: 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.510 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 610121593 0 0) 0x7f1f4c0a1510 con 0x7f1f4c1d31c0 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient(hunting): my global_id is 4347 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f30004ec0 con 0x7f1f4c1d31c0 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 client.?.objecter ms_handle_connect 0x7f1f4c073d40 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1006443240 0 0) 0x7f1f4c140920 con 0x7f1f4c0a5810 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 >> v1:172.21.15.46:6789/0 conn(0x7f1f4c073d40 legacy=0x7f1f4c13cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 >> v1:172.21.15.45:6789/0 conn(0x7f1f4c1d31c0 legacy=0x7f1f4c1d55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 monclient: found mon.c 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4c140360 con 0x7f1f4c0a5810 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1f3c004b40 con 0x7f1f4c0a5810 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1f3c004f90 con 0x7f1f4c0a5810 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:02.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f42ffd640 10 monclient: dump: 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f50bef640 4 set_mon_vals no callback set 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 5 monclient: authenticate success, global_id 4373 2024-09-16T03:05:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient: _renew_subs 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.494+0000 7f1f54656640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1f4c140640 con 0x7f1f4c0a5810 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f1f3c011000 con 0x7f1f4c0a5810 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 10 monclient: _renew_subs 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1f4c1d6cd0 con 0x7f1f4c0a5810 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 1 librados: init done 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7f1f3c04a230 con 0x7f1f4c0a5810 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f42ffd640 10 client.4373.objecter ms_dispatch 0x7f1f4c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f42ffd640 3 client.4373.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f42ffd640 3 client.4373.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f42ffd640 20 client.4373.objecter dump_active .. 0 homeless 2024-09-16T03:05:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.498+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f4c0a23d0 con 0x7f1f4c0a5810 2024-09-16T03:05:02.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.502+0000 7f1f523f2640 10 monclient: get_auth_request con 0x7f1f3003ddc0 auth_method 0 2024-09-16T03:05:02.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.502+0000 7f1f42ffd640 10 client.4373.objecter ms_handle_connect 0x7f1f3003ddc0 2024-09-16T03:05:02.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.502+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1f3c016d90 con 0x7f1f4c0a5810 2024-09-16T03:05:02.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.502+0000 7f1f42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.502+0000 7f1f42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.546+0000 7f7737e80640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:05:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.546+0000 7f7737e80640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:05:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.546+0000 7f7737e80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.546+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7730076b20 con 0x7f77300a5810 2024-09-16T03:05:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.546+0000 7f77267fc640 1 -- 172.21.15.45:0/1166081224 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 203001719) 0x7f7728015b00 con 0x7f77300a5810 2024-09-16T03:05:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.546+0000 7f77267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:05:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.546+0000 7f77267fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:02.551 INFO:teuthology.orchestra.run.smithi045.stdout:51539607556 2024-09-16T03:05:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 10 librados: watch_flush enter 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 10 librados: watch_flush exit 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 20 client.4349.objecter shutdown clearing up homeless session... 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 10 client.4349.objecter successfully canceled tick 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 >> v1:172.21.15.46:6812/149360280 conn(0x7f771403e3d0 legacy=0x7f7714040890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 10 monclient: shutdown 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 >> v1:172.21.15.45:6790/0 conn(0x7f77300a5810 legacy=0x7f77301d9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 shutdown_connections 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 >> 172.21.15.45:0/1166081224 conn(0x7f77300aa720 msgr2=0x7f7730141560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 shutdown_connections 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 1 -- 172.21.15.45:0/1166081224 wait complete. 2024-09-16T03:05:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.550+0000 7f7737e80640 1 librados: shutdown 2024-09-16T03:05:02.575 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.1 2024-09-16T03:05:02.575 DEBUG:teuthology.parallel:result is None 2024-09-16T03:05:02.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.630+0000 7f4719ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:05:02.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.630+0000 7f4719ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:05:02.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.630+0000 7f4719ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.630+0000 7f4719ffb640 1 -- 172.21.15.45:0/764204789 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f470c005380 con 0x7f47340a4d20 2024-09-16T03:05:02.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.630+0000 7f471bfff640 1 -- 172.21.15.45:0/764204789 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 1444633209) 0x7f4728015150 con 0x7f47340a4d20 2024-09-16T03:05:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.630+0000 7f471bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:05:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.630+0000 7f471bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:02.637 INFO:teuthology.orchestra.run.smithi045.stdout:47244640260 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 10 librados: watch_flush enter 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 10 librados: watch_flush exit 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 20 client.4343.objecter shutdown clearing up homeless session... 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 10 client.4343.objecter successfully canceled tick 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 1 -- 172.21.15.45:0/764204789 >> v1:172.21.15.46:6812/149360280 conn(0x7f472003e270 legacy=0x7f4720040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 10 monclient: shutdown 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 1 -- 172.21.15.45:0/764204789 >> v1:172.21.15.45:6790/0 conn(0x7f47340a4d20 legacy=0x7f47341cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 1 -- 172.21.15.45:0/764204789 shutdown_connections 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 1 -- 172.21.15.45:0/764204789 >> 172.21.15.45:0/764204789 conn(0x7f47340aa720 msgr2=0x7f47340a2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 1 -- 172.21.15.45:0/764204789 shutdown_connections 2024-09-16T03:05:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 1 -- 172.21.15.45:0/764204789 wait complete. 2024-09-16T03:05:02.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.634+0000 7f4719ffb640 1 librados: shutdown 2024-09-16T03:05:02.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-09-16T03:05:02.660 DEBUG:teuthology.parallel:result is None 2024-09-16T03:05:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.678+0000 7f0a65983640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:05:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.678+0000 7f0a65983640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:05:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.678+0000 7f0a65983640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.678+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0a60076b20 con 0x7f0a60073d40 2024-09-16T03:05:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.678+0000 7f0a5cff9640 1 -- 172.21.15.45:0/2703235033 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 203001719) 0x7f0a50001810 con 0x7f0a60073d40 2024-09-16T03:05:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.678+0000 7f0a5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:05:02.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.678+0000 7f0a5cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:02.685 INFO:teuthology.orchestra.run.smithi045.stdout:51539607556 2024-09-16T03:05:02.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 10 librados: watch_flush enter 2024-09-16T03:05:02.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 10 librados: watch_flush exit 2024-09-16T03:05:02.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 20 client.4355.objecter shutdown clearing up homeless session... 2024-09-16T03:05:02.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 10 client.4355.objecter successfully canceled tick 2024-09-16T03:05:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 >> v1:172.21.15.46:6812/149360280 conn(0x7f0a4003e0d0 legacy=0x7f0a40040590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 10 monclient: shutdown 2024-09-16T03:05:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 >> v1:172.21.15.45:6790/0 conn(0x7f0a60073d40 legacy=0x7f0a60137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 shutdown_connections 2024-09-16T03:05:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 >> 172.21.15.45:0/2703235033 conn(0x7f0a600aa720 msgr2=0x7f0a600a3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 shutdown_connections 2024-09-16T03:05:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 1 -- 172.21.15.45:0/2703235033 wait complete. 2024-09-16T03:05:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f0a65983640 1 librados: shutdown 2024-09-16T03:05:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f1f54656640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:05:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f1f54656640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:05:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f1f54656640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f1f4c09d6d0 con 0x7f1f4c0a5810 2024-09-16T03:05:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f1f42ffd640 1 -- 172.21.15.45:0/1389594244 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 532354816) 0x7f1f3c01b110 con 0x7f1f4c0a5810 2024-09-16T03:05:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f1f42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:05:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.682+0000 7f1f42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:02.689 INFO:teuthology.orchestra.run.smithi045.stdout:51539607557 2024-09-16T03:05:02.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 10 librados: watch_flush enter 2024-09-16T03:05:02.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 10 librados: watch_flush exit 2024-09-16T03:05:02.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 20 client.4373.objecter shutdown clearing up homeless session... 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 10 client.4373.objecter successfully canceled tick 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 >> v1:172.21.15.46:6812/149360280 conn(0x7f1f3003ddc0 legacy=0x7f1f30040280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 10 monclient: shutdown 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 >> v1:172.21.15.45:6790/0 conn(0x7f1f4c0a5810 legacy=0x7f1f4c13d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 shutdown_connections 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 >> 172.21.15.45:0/1389594244 conn(0x7f1f4c0aa720 msgr2=0x7f1f4c0a3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 shutdown_connections 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 1 -- 172.21.15.45:0/1389594244 wait complete. 2024-09-16T03:05:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.686+0000 7f1f54656640 1 librados: shutdown 2024-09-16T03:05:02.707 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.4 2024-09-16T03:05:02.707 DEBUG:teuthology.parallel:result is None 2024-09-16T03:05:02.711 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.5 2024-09-16T03:05:02.711 DEBUG:teuthology.parallel:result is None 2024-09-16T03:05:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.710+0000 7f866a67c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:05:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.710+0000 7f866a67c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:05:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.710+0000 7f866a67c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.710+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8648005380 con 0x7f8664073d40 2024-09-16T03:05:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.710+0000 7f86617fa640 1 -- 172.21.15.45:0/5069588 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1169801742) 0x7f86540163a0 con 0x7f8664073d40 2024-09-16T03:05:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.710+0000 7f86617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:05:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.710+0000 7f86617fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:02.717 INFO:teuthology.orchestra.run.smithi045.stdout:47244640261 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 10 librados: watch_flush enter 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 10 librados: watch_flush exit 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 20 client.4335.objecter shutdown clearing up homeless session... 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 10 client.4335.objecter successfully canceled tick 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 >> v1:172.21.15.46:6812/149360280 conn(0x7f864403e320 legacy=0x7f86440407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 10 monclient: shutdown 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 >> v1:172.21.15.45:6789/0 conn(0x7f8664073d40 legacy=0x7f8664137ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 shutdown_connections 2024-09-16T03:05:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 >> 172.21.15.45:0/5069588 conn(0x7f86640aa720 msgr2=0x7f86640a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 shutdown_connections 2024-09-16T03:05:02.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 1 -- 172.21.15.45:0/5069588 wait complete. 2024-09-16T03:05:02.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.714+0000 7f866a67c640 1 librados: shutdown 2024-09-16T03:05:02.733 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.0 2024-09-16T03:05:02.733 DEBUG:teuthology.parallel:result is None 2024-09-16T03:05:02.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:05:02.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:05:02.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb8a0005380 con 0x7fb8c00a5810 2024-09-16T03:05:02.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8a77fe640 1 -- 172.21.15.45:0/1651412244 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 203001719) 0x7fb8b8016330 con 0x7fb8c00a5810 2024-09-16T03:05:02.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:05:02.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:02.761 INFO:teuthology.orchestra.run.smithi045.stdout:51539607556 2024-09-16T03:05:02.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 10 librados: watch_flush enter 2024-09-16T03:05:02.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 10 librados: watch_flush exit 2024-09-16T03:05:02.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 20 client.4341.objecter shutdown clearing up homeless session... 2024-09-16T03:05:02.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 10 client.4341.objecter successfully canceled tick 2024-09-16T03:05:02.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 >> v1:172.21.15.46:6812/149360280 conn(0x7fb8ac03e0a0 legacy=0x7fb8ac040560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 10 monclient: shutdown 2024-09-16T03:05:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 >> v1:172.21.15.45:6789/0 conn(0x7fb8c00a5810 legacy=0x7fb8c01d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 shutdown_connections 2024-09-16T03:05:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 >> 172.21.15.45:0/1651412244 conn(0x7fb8c00aa720 msgr2=0x7fb8c00a41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 shutdown_connections 2024-09-16T03:05:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 1 -- 172.21.15.45:0/1651412244 wait complete. 2024-09-16T03:05:02.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.758+0000 7fb8c5697640 1 librados: shutdown 2024-09-16T03:05:02.777 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.3 2024-09-16T03:05:02.777 DEBUG:teuthology.parallel:result is None 2024-09-16T03:05:02.777 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:05:02.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:05:02.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.922+0000 7fb1bfeb4640 10 monclient: get_monmap_and_config 2024-09-16T03:05:02.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.922+0000 7fb1bfeb4640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.922+0000 7fb1bfeb4640 10 monclient: monmap: 2024-09-16T03:05:02.928 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.928 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.928 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:02.927137+0000 2024-09-16T03:05:02.928 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:02.927137+0000 2024-09-16T03:05:02.928 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.928 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.928 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.928 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:02.929 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:02.929 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.922+0000 7fb1bfeb4640 1 Processor -- start 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.922+0000 7fb1bfeb4640 1 -- start start 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.922+0000 7fb1bfeb4640 10 monclient: init 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b8135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): picked mon.noname-c con 0x7fb1b809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): picked mon.noname-a con 0x7fb1b813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b809f7e0 con 0x7fb1b813c010 2024-09-16T03:05:02.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b809b160 con 0x7fb1b809fc40 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b809b440 con 0x7fb1b8135950 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): authenticate will time out at 993.576412s 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bdc50640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb1b8135950 0x7fb1b8135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35276/0 (socket says 172.21.15.45:35276) 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1be451640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb1b813c010 0x7fb1b813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42570/0 (socket says 172.21.15.45:42570) 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bdc50640 1 -- 172.21.15.45:0/1646701482 learned_addr learned my addr 172.21.15.45:0/1646701482 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298811577 0 0) 0x7fb1b809b440 con 0x7fb1b8135950 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient(hunting): my global_id is 4376 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1a0003530 con 0x7fb1b8135950 2024-09-16T03:05:02.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 882323938 0 0) 0x7fb1b809f7e0 con 0x7fb1b813c010 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient(hunting): my global_id is 4350 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1b809b440 con 0x7fb1b813c010 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 927713281 0 0) 0x7fb1b809b440 con 0x7fb1b813c010 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 >> v1:172.21.15.45:6790/0 conn(0x7fb1b8135950 legacy=0x7fb1b8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 >> v1:172.21.15.46:6789/0 conn(0x7fb1b809fc40 legacy=0x7fb1b813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 monclient: found mon.noname-a 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b8136440 con 0x7fb1b813c010 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb1a8001680 con 0x7fb1b813c010 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: dump: 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.941 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.942 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.942 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.942 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.942 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.942 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.942 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 5 monclient: authenticate success, global_id 4350 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1a8001820 con 0x7fb1b813c010 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 1 -- 172.21.15.45:0/1646701482 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb1a8001cb0 con 0x7fb1b813c010 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bcc4e640 10 monclient: dump: 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.943 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 4 set_mon_vals no callback set 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient: shutdown 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1646701482 >> v1:172.21.15.45:6789/0 conn(0x7fb1b813c010 legacy=0x7fb1b813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1646701482 shutdown_connections 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1646701482 >> 172.21.15.45:0/1646701482 conn(0x7fb1b80a2bd0 msgr2=0x7fb1b80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1646701482 shutdown_connections 2024-09-16T03:05:02.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1646701482 wait complete. 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient: build_initial_monmap 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient: monmap: 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:02.945 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 librados: starting msgr at 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 librados: starting objecter 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 Processor -- start 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- start start 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 librados: setting wanted keys 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 librados: calling monclient init 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient: init 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:02.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): picked mon.noname-c con 0x7fb1b8135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): picked mon.noname-a con 0x7fb1b813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b80718e0 con 0x7fb1b813c010 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b813bb90 con 0x7fb1b809fc40 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b813b6f0 con 0x7fb1b8135950 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bfeb4640 10 monclient(hunting): authenticate will time out at 993.579415s 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bd44f640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb1b8135950 0x7fb1b806ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35282/0 (socket says 172.21.15.45:35282) 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1bd44f640 1 -- 172.21.15.45:0/1141535473 learned_addr learned my addr 172.21.15.45:0/1141535473 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 client.?.objecter ms_handle_connect 0x7fb1b8135950 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 client.?.objecter ms_handle_connect 0x7fb1b813c010 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3535342054 0 0) 0x7fb1b813b6f0 con 0x7fb1b8135950 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 monclient(hunting): my global_id is 4379 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1a4004ef0 con 0x7fb1b8135950 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778394134 0 0) 0x7fb1b80718e0 con 0x7fb1b813c010 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 monclient(hunting): my global_id is 4353 2024-09-16T03:05:02.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1b813b6f0 con 0x7fb1b813c010 2024-09-16T03:05:02.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 client.?.objecter ms_handle_connect 0x7fb1b809fc40 2024-09-16T03:05:02.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:02.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1357674807 0 0) 0x7fb1a4004ef0 con 0x7fb1b8135950 2024-09-16T03:05:02.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1b80718e0 con 0x7fb1b8135950 2024-09-16T03:05:02.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.926+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1442660929 0 0) 0x7fb1b813b6f0 con 0x7fb1b813c010 2024-09-16T03:05:02.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1a4004ef0 con 0x7fb1b813c010 2024-09-16T03:05:02.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb1ac0019a0 con 0x7fb1b8135950 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): dump: 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.950 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.952 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb1a8001b70 con 0x7fb1b813c010 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): dump: 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626741593 0 0) 0x7fb1b813bb90 con 0x7fb1b809fc40 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient(hunting): my global_id is 4372 2024-09-16T03:05:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1b813b6f0 con 0x7fb1b809fc40 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1074662503 0 0) 0x7fb1b80718e0 con 0x7fb1b8135950 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 >> v1:172.21.15.46:6789/0 conn(0x7fb1b809fc40 legacy=0x7fb1b806e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 >> v1:172.21.15.45:6789/0 conn(0x7fb1b813c010 legacy=0x7fb1b806f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 monclient: found mon.c 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b809d2c0 con 0x7fb1b8135950 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 5 monclient: authenticate success, global_id 4379 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 10 monclient: _renew_subs 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b8071bc0 con 0x7fb1b8135950 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 10 monclient: _renew_subs 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1b81d3b90 con 0x7fb1b8135950 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 1 librados: init done 2024-09-16T03:05:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1ac001ce0 con 0x7fb1b8135950 2024-09-16T03:05:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb1ac003a70 con 0x7fb1b8135950 2024-09-16T03:05:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b7fff640 4 set_mon_vals no callback set 2024-09-16T03:05:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 monclient: dump: 2024-09-16T03:05:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb198005090 con 0x7fb1b8135950 2024-09-16T03:05:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fb1ac003c10 con 0x7fb1b8135950 2024-09-16T03:05:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1393625185 0 0) 0x7fb1ac04b0c0 con 0x7fb1b8135950 2024-09-16T03:05:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 client.4379.objecter ms_dispatch 0x7fb1b8000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:05:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 3 client.4379.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:05:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 3 client.4379.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:05:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 20 client.4379.objecter dump_active .. 0 homeless 2024-09-16T03:05:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1bdc50640 10 monclient: get_auth_request con 0x7fb1a403e390 auth_method 0 2024-09-16T03:05:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.930+0000 7fb1b67fc640 10 client.4379.objecter ms_handle_connect 0x7fb1a403e390 2024-09-16T03:05:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.934+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fb1ac016a80 con 0x7fb1b8135950 2024-09-16T03:05:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.934+0000 7fb1b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:02.934+0000 7fb1b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.094+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb198000ba0 con 0x7fb1a403e390 2024-09-16T03:05:03.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1b67fc640 1 -- 172.21.15.45:0/1141535473 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34147 (unknown 2965378022 0 1958187928) 0x7fb198000ba0 con 0x7fb1a403e390 2024-09-16T03:05:03.101 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:05:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-16T03:05:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 10 librados: watch_flush enter 2024-09-16T03:05:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 10 librados: watch_flush exit 2024-09-16T03:05:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 20 client.4379.objecter shutdown clearing up homeless session... 2024-09-16T03:05:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 10 client.4379.objecter successfully canceled tick 2024-09-16T03:05:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 >> v1:172.21.15.46:6812/149360280 conn(0x7fb1a403e390 legacy=0x7fb1a4040850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 10 monclient: shutdown 2024-09-16T03:05:03.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:03.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 >> v1:172.21.15.45:6790/0 conn(0x7fb1b8135950 legacy=0x7fb1b806ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 shutdown_connections 2024-09-16T03:05:03.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 >> 172.21.15.45:0/1141535473 conn(0x7fb1b80a2bd0 msgr2=0x7fb1b813eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:03.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 shutdown_connections 2024-09-16T03:05:03.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 1 -- 172.21.15.45:0/1141535473 wait complete. 2024-09-16T03:05:03.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.098+0000 7fb1bfeb4640 1 librados: shutdown 2024-09-16T03:05:03.118 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T03:05:01.460079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":185008,"kb_used_data":2008,"kb_used_omap":9,"kb_used_meta":182838,"kb_avail":566046032,"statfs":{"total":579820584960,"available":579631136768,"internally_reserved":0,"allocated":2056192,"data_stored":1369846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":187226786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.357930"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.132050+0000","last_change":"2024-09-16T03:04:59.132718+0000","last_active":"2024-09-16T03:04:59.132050+0000","last_peered":"2024-09-16T03:04:59.132050+0000","last_clean":"2024-09-16T03:04:59.132050+0000","last_became_active":"2024-09-16T03:04:57.691813+0000","last_became_peered":"2024-09-16T03:04:57.691813+0000","last_unstale":"2024-09-16T03:04:59.132050+0000","last_undegraded":"2024-09-16T03:04:59.132050+0000","last_fullsized":"2024-09-16T03:04:59.132050+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:55.115695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015712199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.142318+0000","last_change":"2024-09-16T03:04:59.142318+0000","last_active":"2024-09-16T03:04:59.142318+0000","last_peered":"2024-09-16T03:04:59.142318+0000","last_clean":"2024-09-16T03:04:59.142318+0000","last_became_active":"2024-09-16T03:04:57.117464+0000","last_became_peered":"2024-09-16T03:04:57.117464+0000","last_unstale":"2024-09-16T03:04:59.142318+0000","last_undegraded":"2024-09-16T03:04:59.142318+0000","last_fullsized":"2024-09-16T03:04:59.142318+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:54:40.258398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012748499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.131459+0000","last_change":"2024-09-16T03:04:59.131904+0000","last_active":"2024-09-16T03:04:59.131459+0000","last_peered":"2024-09-16T03:04:59.131459+0000","last_clean":"2024-09-16T03:04:59.131459+0000","last_became_active":"2024-09-16T03:04:57.113180+0000","last_became_peered":"2024-09-16T03:04:57.113180+0000","last_unstale":"2024-09-16T03:04:59.131459+0000","last_undegraded":"2024-09-16T03:04:59.131459+0000","last_fullsized":"2024-09-16T03:04:59.131459+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:30.278408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087237100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.142273+0000","last_change":"2024-09-16T03:04:59.142273+0000","last_active":"2024-09-16T03:04:59.142273+0000","last_peered":"2024-09-16T03:04:59.142273+0000","last_clean":"2024-09-16T03:04:59.142273+0000","last_became_active":"2024-09-16T03:04:57.117660+0000","last_became_peered":"2024-09-16T03:04:57.117660+0000","last_unstale":"2024-09-16T03:04:59.142273+0000","last_undegraded":"2024-09-16T03:04:59.142273+0000","last_fullsized":"2024-09-16T03:04:59.142273+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:22.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.138496+0000","last_change":"2024-09-16T03:04:59.138496+0000","last_active":"2024-09-16T03:04:59.138496+0000","last_peered":"2024-09-16T03:04:59.138496+0000","last_clean":"2024-09-16T03:04:59.138496+0000","last_became_active":"2024-09-16T03:04:57.119695+0000","last_became_peered":"2024-09-16T03:04:57.119695+0000","last_unstale":"2024-09-16T03:04:59.138496+0000","last_undegraded":"2024-09-16T03:04:59.138496+0000","last_fullsized":"2024-09-16T03:04:59.138496+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:24.631318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024209740000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.132613+0000","last_change":"2024-09-16T03:04:59.133304+0000","last_active":"2024-09-16T03:04:59.132613+0000","last_peered":"2024-09-16T03:04:59.132613+0000","last_clean":"2024-09-16T03:04:59.132613+0000","last_became_active":"2024-09-16T03:04:57.118413+0000","last_became_peered":"2024-09-16T03:04:57.118413+0000","last_unstale":"2024-09-16T03:04:59.132613+0000","last_undegraded":"2024-09-16T03:04:59.132613+0000","last_fullsized":"2024-09-16T03:04:59.132613+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:24.300841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014587720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.131619+0000","last_change":"2024-09-16T03:04:59.132034+0000","last_active":"2024-09-16T03:04:59.131619+0000","last_peered":"2024-09-16T03:04:59.131619+0000","last_clean":"2024-09-16T03:04:59.131619+0000","last_became_active":"2024-09-16T03:04:57.117048+0000","last_became_peered":"2024-09-16T03:04:57.117048+0000","last_unstale":"2024-09-16T03:04:59.131619+0000","last_undegraded":"2024-09-16T03:04:59.131619+0000","last_fullsized":"2024-09-16T03:04:59.131619+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:57.899748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099232000000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.136747+0000","last_change":"2024-09-16T03:04:59.137133+0000","last_active":"2024-09-16T03:04:59.136747+0000","last_peered":"2024-09-16T03:04:59.136747+0000","last_clean":"2024-09-16T03:04:59.136747+0000","last_became_active":"2024-09-16T03:04:57.116849+0000","last_became_peered":"2024-09-16T03:04:57.116849+0000","last_unstale":"2024-09-16T03:04:59.136747+0000","last_undegraded":"2024-09-16T03:04:59.136747+0000","last_fullsized":"2024-09-16T03:04:59.136747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:51.385856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00103192,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.130859+0000","last_change":"2024-09-16T03:04:51.319467+0000","last_active":"2024-09-16T03:04:59.130859+0000","last_peered":"2024-09-16T03:04:59.130859+0000","last_clean":"2024-09-16T03:04:59.130859+0000","last_became_active":"2024-09-16T03:04:51.317160+0000","last_became_peered":"2024-09-16T03:04:51.317160+0000","last_unstale":"2024-09-16T03:04:59.130859+0000","last_undegraded":"2024-09-16T03:04:59.130859+0000","last_fullsized":"2024-09-16T03:04:59.130859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_clean_scrub_stamp":"2024-09-16T03:04:49.540253+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:15.016218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:05:03.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.226+0000 7f1979fc4640 10 monclient: get_monmap_and_config 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.226+0000 7f1979fc4640 10 monclient: build_initial_monmap 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.226+0000 7f1979fc4640 10 monclient: monmap: 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:03.231429+0000 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:03.231429+0000 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.232 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:03.233 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:03.233 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:03.233 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 Processor -- start 2024-09-16T03:05:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- start start 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: init 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): picked mon.noname-c con 0x7f19740a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): picked mon.noname-b con 0x7f19740a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): picked mon.noname-a con 0x7f1974073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197409d9e0 con 0x7f1974073d40 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197409dc20 con 0x7f19740a4d20 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1974073360 con 0x7f19740a5810 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): authenticate will time out at 993.880998s 2024-09-16T03:05:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1973fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1974073d40 0x7f1974140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42592/0 (socket says 172.21.15.45:42592) 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1973fff640 1 -- 172.21.15.45:0/2403327162 learned_addr learned my addr 172.21.15.45:0/2403327162 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3281861454 0 0) 0x7f197409d9e0 con 0x7f1974073d40 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient(hunting): my global_id is 4356 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1954003530 con 0x7f1974073d40 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3885510072 0 0) 0x7f1974073360 con 0x7f19740a5810 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient(hunting): my global_id is 4382 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f197409d9e0 con 0x7f19740a5810 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4250523449 0 0) 0x7f197409d9e0 con 0x7f19740a5810 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 >> v1:172.21.15.46:6789/0 conn(0x7f19740a4d20 legacy=0x7f19740a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 >> v1:172.21.15.45:6789/0 conn(0x7f1974073d40 legacy=0x7f1974140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 monclient: found mon.noname-b 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197409cf30 con 0x7f19740a5810 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: _finish_auth 0 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f195c001680 con 0x7f19740a5810 2024-09-16T03:05:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 5 monclient: authenticate success, global_id 4382 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: dump: 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f195c001820 con 0x7f19740a5810 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 1 -- 172.21.15.45:0/2403327162 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f195c001c30 con 0x7f19740a5810 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1972ffd640 10 monclient: dump: 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 4 set_mon_vals no callback set 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: shutdown 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- 172.21.15.45:0/2403327162 >> v1:172.21.15.45:6790/0 conn(0x7f19740a5810 legacy=0x7f197413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- 172.21.15.45:0/2403327162 shutdown_connections 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- 172.21.15.45:0/2403327162 >> 172.21.15.45:0/2403327162 conn(0x7f19740aa720 msgr2=0x7f19740aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- 172.21.15.45:0/2403327162 shutdown_connections 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- 172.21.15.45:0/2403327162 wait complete. 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: build_initial_monmap 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: monmap: 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 librados: starting msgr at 2024-09-16T03:05:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 librados: starting objecter 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 Processor -- start 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- start start 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 librados: setting wanted keys 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 librados: calling monclient init 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: init 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): picked mon.noname-c con 0x7f1974073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): picked mon.noname-a con 0x7f19740a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): picked mon.noname-b con 0x7f19740a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19741401e0 con 0x7f19740a4d20 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19741404c0 con 0x7f19740a5810 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197413d8b0 con 0x7f1974073d40 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1979fc4640 10 monclient(hunting): authenticate will time out at 993.883716s 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f196bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f19740a4d20 0x7f197413d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42608/0 (socket says 172.21.15.45:42608) 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f19737fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1974073d40 0x7f197413ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35306/0 (socket says 172.21.15.45:35306) 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f196bfff640 1 -- 172.21.15.45:0/3936351018 learned_addr learned my addr 172.21.15.45:0/3936351018 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1970ff9640 10 client.?.objecter ms_handle_connect 0x7f19740a4d20 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1970ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1970ff9640 10 client.?.objecter ms_handle_connect 0x7f1974073d40 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.230+0000 7f1970ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3009652107 0 0) 0x7f197413d8b0 con 0x7f1974073d40 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): my global_id is 4385 2024-09-16T03:05:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1958004ec0 con 0x7f1974073d40 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 972213880 0 0) 0x7f19741401e0 con 0x7f19740a4d20 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): my global_id is 4359 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f197413d8b0 con 0x7f19740a4d20 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 client.?.objecter ms_handle_connect 0x7f19740a5810 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1476348365 0 0) 0x7f197413d8b0 con 0x7f19740a4d20 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19741401e0 con 0x7f19740a4d20 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f195c001680 con 0x7f19740a4d20 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): dump: 2024-09-16T03:05:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 675152604 0 0) 0x7f1958004ec0 con 0x7f1974073d40 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f197413d8b0 con 0x7f1974073d40 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1964001920 con 0x7f1974073d40 2024-09-16T03:05:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): dump: 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264992101 0 0) 0x7f19741404c0 con 0x7f19740a5810 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient(hunting): my global_id is 4378 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1958004ec0 con 0x7f19740a5810 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3384610709 0 0) 0x7f19741401e0 con 0x7f19740a4d20 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 >> v1:172.21.15.45:6790/0 conn(0x7f1974073d40 legacy=0x7f197413ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 >> v1:172.21.15.46:6789/0 conn(0x7f19740a5810 legacy=0x7f19741d51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 monclient: found mon.a 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19740a1510 con 0x7f19740a4d20 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient: _finish_auth 0 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:05:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 5 monclient: authenticate success, global_id 4359 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 10 monclient: _renew_subs 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f197413f930 con 0x7f19740a4d20 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 10 monclient: _renew_subs 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f19741407a0 con 0x7f19740a4d20 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 1 librados: init done 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f195c001820 con 0x7f19740a4d20 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f195c003b10 con 0x7f19740a4d20 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f19727fc640 4 set_mon_vals no callback set 2024-09-16T03:05:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 monclient: dump: 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194c005090 con 0x7f19740a4d20 2024-09-16T03:05:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f195c003cb0 con 0x7f19740a4d20 2024-09-16T03:05:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f195c04b0b0 con 0x7f19740a4d20 2024-09-16T03:05:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 client.4359.objecter ms_dispatch 0x7f1974000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 3 client.4359.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 3 client.4359.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 20 client.4359.objecter dump_active .. 0 homeless 2024-09-16T03:05:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f19737fe640 10 monclient: get_auth_request con 0x7f195803e220 auth_method 0 2024-09-16T03:05:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.234+0000 7f1970ff9640 10 client.4359.objecter ms_handle_connect 0x7f195803e220 2024-09-16T03:05:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.238+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f195c016b30 con 0x7f19740a4d20 2024-09-16T03:05:03.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.238+0000 7f1970ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:03.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.238+0000 7f1970ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:03.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.398+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f194c000ba0 con 0x7f195803e220 2024-09-16T03:05:03.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1970ff9640 1 -- 172.21.15.45:0/3936351018 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34147 (unknown 2965378022 0 4201965849) 0x7f194c000ba0 con 0x7f195803e220 2024-09-16T03:05:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:05:03.406 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-16T03:05:03.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 10 librados: watch_flush enter 2024-09-16T03:05:03.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 10 librados: watch_flush exit 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 20 client.4359.objecter shutdown clearing up homeless session... 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 10 client.4359.objecter successfully canceled tick 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 >> v1:172.21.15.46:6812/149360280 conn(0x7f195803e220 legacy=0x7f19580406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 10 monclient: shutdown 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 >> v1:172.21.15.45:6789/0 conn(0x7f19740a4d20 legacy=0x7f197413d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 shutdown_connections 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 >> 172.21.15.45:0/3936351018 conn(0x7f19740aa720 msgr2=0x7f19740a4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 shutdown_connections 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 1 -- 172.21.15.45:0/3936351018 wait complete. 2024-09-16T03:05:03.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.402+0000 7f1979fc4640 1 librados: shutdown 2024-09-16T03:05:03.422 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T03:05:03.163934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":185008,"kb_used_data":2008,"kb_used_omap":9,"kb_used_meta":182838,"kb_avail":566046032,"statfs":{"total":579820584960,"available":579631136768,"internally_reserved":0,"allocated":2056192,"data_stored":1369846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":187226786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.059280"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.132050+0000","last_change":"2024-09-16T03:04:59.132718+0000","last_active":"2024-09-16T03:04:59.132050+0000","last_peered":"2024-09-16T03:04:59.132050+0000","last_clean":"2024-09-16T03:04:59.132050+0000","last_became_active":"2024-09-16T03:04:57.691813+0000","last_became_peered":"2024-09-16T03:04:57.691813+0000","last_unstale":"2024-09-16T03:04:59.132050+0000","last_undegraded":"2024-09-16T03:04:59.132050+0000","last_fullsized":"2024-09-16T03:04:59.132050+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:55.115695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015712199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.142318+0000","last_change":"2024-09-16T03:04:59.142318+0000","last_active":"2024-09-16T03:04:59.142318+0000","last_peered":"2024-09-16T03:04:59.142318+0000","last_clean":"2024-09-16T03:04:59.142318+0000","last_became_active":"2024-09-16T03:04:57.117464+0000","last_became_peered":"2024-09-16T03:04:57.117464+0000","last_unstale":"2024-09-16T03:04:59.142318+0000","last_undegraded":"2024-09-16T03:04:59.142318+0000","last_fullsized":"2024-09-16T03:04:59.142318+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:54:40.258398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012748499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.131459+0000","last_change":"2024-09-16T03:04:59.131904+0000","last_active":"2024-09-16T03:04:59.131459+0000","last_peered":"2024-09-16T03:04:59.131459+0000","last_clean":"2024-09-16T03:04:59.131459+0000","last_became_active":"2024-09-16T03:04:57.113180+0000","last_became_peered":"2024-09-16T03:04:57.113180+0000","last_unstale":"2024-09-16T03:04:59.131459+0000","last_undegraded":"2024-09-16T03:04:59.131459+0000","last_fullsized":"2024-09-16T03:04:59.131459+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:30.278408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087237100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.142273+0000","last_change":"2024-09-16T03:04:59.142273+0000","last_active":"2024-09-16T03:04:59.142273+0000","last_peered":"2024-09-16T03:04:59.142273+0000","last_clean":"2024-09-16T03:04:59.142273+0000","last_became_active":"2024-09-16T03:04:57.117660+0000","last_became_peered":"2024-09-16T03:04:57.117660+0000","last_unstale":"2024-09-16T03:04:59.142273+0000","last_undegraded":"2024-09-16T03:04:59.142273+0000","last_fullsized":"2024-09-16T03:04:59.142273+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:22.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.138496+0000","last_change":"2024-09-16T03:04:59.138496+0000","last_active":"2024-09-16T03:04:59.138496+0000","last_peered":"2024-09-16T03:04:59.138496+0000","last_clean":"2024-09-16T03:04:59.138496+0000","last_became_active":"2024-09-16T03:04:57.119695+0000","last_became_peered":"2024-09-16T03:04:57.119695+0000","last_unstale":"2024-09-16T03:04:59.138496+0000","last_undegraded":"2024-09-16T03:04:59.138496+0000","last_fullsized":"2024-09-16T03:04:59.138496+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:24.631318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024209740000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.132613+0000","last_change":"2024-09-16T03:04:59.133304+0000","last_active":"2024-09-16T03:04:59.132613+0000","last_peered":"2024-09-16T03:04:59.132613+0000","last_clean":"2024-09-16T03:04:59.132613+0000","last_became_active":"2024-09-16T03:04:57.118413+0000","last_became_peered":"2024-09-16T03:04:57.118413+0000","last_unstale":"2024-09-16T03:04:59.132613+0000","last_undegraded":"2024-09-16T03:04:59.132613+0000","last_fullsized":"2024-09-16T03:04:59.132613+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:24.300841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014587720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.131619+0000","last_change":"2024-09-16T03:04:59.132034+0000","last_active":"2024-09-16T03:04:59.131619+0000","last_peered":"2024-09-16T03:04:59.131619+0000","last_clean":"2024-09-16T03:04:59.131619+0000","last_became_active":"2024-09-16T03:04:57.117048+0000","last_became_peered":"2024-09-16T03:04:57.117048+0000","last_unstale":"2024-09-16T03:04:59.131619+0000","last_undegraded":"2024-09-16T03:04:59.131619+0000","last_fullsized":"2024-09-16T03:04:59.131619+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:57.899748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099232000000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.136747+0000","last_change":"2024-09-16T03:04:59.137133+0000","last_active":"2024-09-16T03:04:59.136747+0000","last_peered":"2024-09-16T03:04:59.136747+0000","last_clean":"2024-09-16T03:04:59.136747+0000","last_became_active":"2024-09-16T03:04:57.116849+0000","last_became_peered":"2024-09-16T03:04:57.116849+0000","last_unstale":"2024-09-16T03:04:59.136747+0000","last_undegraded":"2024-09-16T03:04:59.136747+0000","last_fullsized":"2024-09-16T03:04:59.136747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:51.385856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00103192,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.130859+0000","last_change":"2024-09-16T03:04:51.319467+0000","last_active":"2024-09-16T03:04:59.130859+0000","last_peered":"2024-09-16T03:04:59.130859+0000","last_clean":"2024-09-16T03:04:59.130859+0000","last_became_active":"2024-09-16T03:04:51.317160+0000","last_became_peered":"2024-09-16T03:04:51.317160+0000","last_unstale":"2024-09-16T03:04:59.130859+0000","last_undegraded":"2024-09-16T03:04:59.130859+0000","last_fullsized":"2024-09-16T03:04:59.130859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_clean_scrub_stamp":"2024-09-16T03:04:49.540253+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:15.016218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:05:03.423 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:05:03.424 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T03:05:03.424 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T03:05:03.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T03:05:03.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient: get_monmap_and_config 2024-09-16T03:05:03.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient: build_initial_monmap 2024-09-16T03:05:03.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient: monmap: 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:03.530080+0000 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:03.530080+0000 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:03.531 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 1 Processor -- start 2024-09-16T03:05:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 1 -- start start 2024-09-16T03:05:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient: init 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient(hunting): picked mon.noname-b con 0x7f5514135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient(hunting): picked mon.noname-a con 0x7f551409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient(hunting): picked mon.noname-c con 0x7f551413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551409f7e0 con 0x7f551409fc40 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551409b160 con 0x7f551413c010 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551409b440 con 0x7f5514135950 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5518d49640 10 monclient(hunting): authenticate will time out at 994.179139s 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5511d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f551409fc40 0x7f551413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42610/0 (socket says 172.21.15.45:42610) 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f5511d9b640 1 -- 172.21.15.45:0/1843060809 learned_addr learned my addr 172.21.15.45:0/1843060809 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551259c640 1 --1- 172.21.15.45:0/1843060809 >> v1:172.21.15.45:6790/0 conn(0x7f5514135950 0x7f5514135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35316/0 (socket says 172.21.15.45:35316) 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2546490605 0 0) 0x7f551409f7e0 con 0x7f551409fc40 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient(hunting): my global_id is 4362 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54fc003530 con 0x7f551409fc40 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 64207296 0 0) 0x7f551409b440 con 0x7f5514135950 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient(hunting): my global_id is 4388 2024-09-16T03:05:03.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f551409f7e0 con 0x7f5514135950 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1278977592 0 0) 0x7f551409f7e0 con 0x7f5514135950 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 >> v1:172.21.15.46:6789/0 conn(0x7f551413c010 legacy=0x7f551413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 >> v1:172.21.15.45:6789/0 conn(0x7f551409fc40 legacy=0x7f551413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient: _finish_hunting 0 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 1 monclient: found mon.noname-b 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5514136440 con 0x7f5514135950 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.526+0000 7f551159a640 10 monclient: _finish_auth 0 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 10 monclient: _check_auth_tickets 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5504001680 con 0x7f5514135950 2024-09-16T03:05:03.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 10 monclient: dump: 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 5 monclient: authenticate success, global_id 4388 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5504001820 con 0x7f5514135950 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 1 -- 172.21.15.45:0/1843060809 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5504001cb0 con 0x7f5514135950 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 4 set_mon_vals no callback set 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f551159a640 10 monclient: dump: 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.542 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: shutdown 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- 172.21.15.45:0/1843060809 >> v1:172.21.15.45:6790/0 conn(0x7f5514135950 legacy=0x7f5514135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- 172.21.15.45:0/1843060809 shutdown_connections 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- 172.21.15.45:0/1843060809 >> 172.21.15.45:0/1843060809 conn(0x7f55140a2bd0 msgr2=0x7f55140a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- 172.21.15.45:0/1843060809 shutdown_connections 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- 172.21.15.45:0/1843060809 wait complete. 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: build_initial_monmap 2024-09-16T03:05:03.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: monmap: 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:03.544 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 librados: starting msgr at 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 librados: starting objecter 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 Processor -- start 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- start start 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 librados: setting wanted keys 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 librados: calling monclient init 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: init 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient(hunting): picked mon.noname-b con 0x7f551409fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient(hunting): picked mon.noname-c con 0x7f5514135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient(hunting): picked mon.noname-a con 0x7f551413c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5514071dd0 con 0x7f551413c010 2024-09-16T03:05:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5514070a70 con 0x7f551409fc40 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55140739e0 con 0x7f5514135950 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient(hunting): authenticate will time out at 994.182215s 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5511d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5514135950 0x7f5514070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35320/0 (socket says 172.21.15.45:35320) 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5511d9b640 1 -- 172.21.15.45:0/306376406 learned_addr learned my addr 172.21.15.45:0/306376406 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5512d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f551413c010 0x7f55141d4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42614/0 (socket says 172.21.15.45:42614) 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 client.?.objecter ms_handle_connect 0x7f5514135950 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 client.?.objecter ms_handle_connect 0x7f551413c010 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408405575 0 0) 0x7f55140739e0 con 0x7f5514135950 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): my global_id is 4391 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54f4004ef0 con 0x7f5514135950 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286399138 0 0) 0x7f5514071dd0 con 0x7f551413c010 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): my global_id is 4365 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55140739e0 con 0x7f551413c010 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 client.?.objecter ms_handle_connect 0x7f551409fc40 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 316125924 0 0) 0x7f55140739e0 con 0x7f551413c010 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5514071dd0 con 0x7f551413c010 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2336895828 0 0) 0x7f54f4004ef0 con 0x7f5514135950 2024-09-16T03:05:03.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55140739e0 con 0x7f5514135950 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f550c001c40 con 0x7f551413c010 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): dump: 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.548 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f55080013a0 con 0x7f5514135950 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): dump: 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.549 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2320942928 0 0) 0x7f5514070a70 con 0x7f551409fc40 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient(hunting): my global_id is 4384 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54f4004ef0 con 0x7f551409fc40 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2098295229 0 0) 0x7f55140739e0 con 0x7f5514135950 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 >> v1:172.21.15.46:6789/0 conn(0x7f551409fc40 legacy=0x7f551406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 >> v1:172.21.15.45:6789/0 conn(0x7f551413c010 legacy=0x7f55141d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 monclient: found mon.c 2024-09-16T03:05:03.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551413b6f0 con 0x7f5514135950 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient: _finish_auth 0 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 5 monclient: authenticate success, global_id 4391 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: _renew_subs 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f551413f520 con 0x7f5514135950 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: _renew_subs 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5514073420 con 0x7f5514135950 2024-09-16T03:05:03.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 librados: init done 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5508001540 con 0x7f5514135950 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5508004c40 con 0x7f5514135950 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5510d99640 4 set_mon_vals no callback set 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5502ffd640 10 monclient: dump: 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.552 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.530+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54f8005090 con 0x7f5514135950 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.534+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f5508010cb0 con 0x7f5514135950 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.534+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f550804b010 con 0x7f5514135950 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.534+0000 7f5502ffd640 10 client.4391.objecter ms_dispatch 0x7f5514000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.534+0000 7f5502ffd640 3 client.4391.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.534+0000 7f5502ffd640 3 client.4391.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:03.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.534+0000 7f5502ffd640 20 client.4391.objecter dump_active .. 0 homeless 2024-09-16T03:05:03.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.534+0000 7f551259c640 10 monclient: get_auth_request con 0x7f54f403e270 auth_method 0 2024-09-16T03:05:03.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.534+0000 7f5502ffd640 10 client.4391.objecter ms_handle_connect 0x7f54f403e270 2024-09-16T03:05:03.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.538+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5508016a90 con 0x7f5514135950 2024-09-16T03:05:03.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.538+0000 7f5502ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:03.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.538+0000 7f5502ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:03.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.738+0000 7f5518d49640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T03:05:03.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.738+0000 7f5518d49640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T03:05:03.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.738+0000 7f5518d49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:03.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.738+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f54f8005380 con 0x7f5514135950 2024-09-16T03:05:03.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.738+0000 7f5502ffd640 1 -- 172.21.15.45:0/306376406 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f5508016390 con 0x7f5514135950 2024-09-16T03:05:03.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.738+0000 7f5502ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T03:05:03.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.738+0000 7f5502ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:03.744 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:05:03.745 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T03:05:03.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 10 librados: watch_flush enter 2024-09-16T03:05:03.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 10 librados: watch_flush exit 2024-09-16T03:05:03.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 20 client.4391.objecter shutdown clearing up homeless session... 2024-09-16T03:05:03.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 10 client.4391.objecter successfully canceled tick 2024-09-16T03:05:03.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 >> v1:172.21.15.46:6812/149360280 conn(0x7f54f403e270 legacy=0x7f54f4040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 10 monclient: shutdown 2024-09-16T03:05:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 >> v1:172.21.15.45:6790/0 conn(0x7f5514135950 legacy=0x7f5514070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 shutdown_connections 2024-09-16T03:05:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 >> 172.21.15.45:0/306376406 conn(0x7f55140a2bd0 msgr2=0x7f551413ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 shutdown_connections 2024-09-16T03:05:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 1 -- 172.21.15.45:0/306376406 wait complete. 2024-09-16T03:05:03.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.742+0000 7f5518d49640 1 librados: shutdown 2024-09-16T03:05:03.761 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T03:05:03.762 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T03:05:03.779 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T03:05:03.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:05:03.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.894+0000 7fc35c394640 10 monclient: get_monmap_and_config 2024-09-16T03:05:03.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.894+0000 7fc35c394640 10 monclient: build_initial_monmap 2024-09-16T03:05:03.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.894+0000 7fc35c394640 10 monclient: monmap: 2024-09-16T03:05:03.899 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:03.900 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.900 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:03.899057+0000 2024-09-16T03:05:03.900 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:03.899057+0000 2024-09-16T03:05:03.900 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:03.900 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.900 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:03.900 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:03.900 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:03.900 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.894+0000 7fc35c394640 1 Processor -- start 2024-09-16T03:05:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.894+0000 7fc35c394640 1 -- start start 2024-09-16T03:05:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.894+0000 7fc35c394640 10 monclient: init 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): picked mon.noname-b con 0x7fc35409e420 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): picked mon.noname-c con 0x7fc35409e800 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): picked mon.noname-a con 0x7fc35409ec10 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3541386b0 con 0x7fc35409ec10 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc35406e590 con 0x7fc35409e800 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc354075120 con 0x7fc35409e420 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): authenticate will time out at 994.548450s 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35a130640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc35409e420 0x7fc354137fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35326/0 (socket says 172.21.15.45:35326) 2024-09-16T03:05:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35a931640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc35409ec10 0x7fc3541404b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42628/0 (socket says 172.21.15.45:42628) 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35a130640 1 -- 172.21.15.45:0/3575611041 learned_addr learned my addr 172.21.15.45:0/3575611041 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515708360 0 0) 0x7fc3541386b0 con 0x7fc35409ec10 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): my global_id is 4368 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc340003530 con 0x7fc35409ec10 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997622288 0 0) 0x7fc354075120 con 0x7fc35409e420 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): my global_id is 4394 2024-09-16T03:05:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3541386b0 con 0x7fc35409e420 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2243142205 0 0) 0x7fc350001120 con 0x7fc35409e800 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient(hunting): my global_id is 4387 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc354075120 con 0x7fc35409e800 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2547957553 0 0) 0x7fc340003530 con 0x7fc35409ec10 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 >> v1:172.21.15.45:6790/0 conn(0x7fc35409e420 legacy=0x7fc354137fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 >> v1:172.21.15.46:6789/0 conn(0x7fc35409e800 legacy=0x7fc35413cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: _finish_hunting 0 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 monclient: found mon.noname-a 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:03.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35406e2b0 con 0x7fc35409ec10 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: _finish_auth 0 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: _check_auth_tickets 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc34c001a20 con 0x7fc35409ec10 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: dump: 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.907 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 5 monclient: authenticate success, global_id 4368 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc34c001bc0 con 0x7fc35409ec10 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 1 -- 172.21.15.45:0/3575611041 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc34c001fd0 con 0x7fc35409ec10 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:03.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35912e640 10 monclient: dump: 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:03.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 4 set_mon_vals no callback set 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient: shutdown 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- 172.21.15.45:0/3575611041 >> v1:172.21.15.45:6789/0 conn(0x7fc35409ec10 legacy=0x7fc3541404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- 172.21.15.45:0/3575611041 shutdown_connections 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- 172.21.15.45:0/3575611041 >> 172.21.15.45:0/3575611041 conn(0x7fc3540a4ff0 msgr2=0x7fc3540a5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- 172.21.15.45:0/3575611041 shutdown_connections 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- 172.21.15.45:0/3575611041 wait complete. 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient: build_initial_monmap 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient: monmap: 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 librados: starting msgr at 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 librados: starting objecter 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 Processor -- start 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- start start 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 librados: setting wanted keys 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 librados: calling monclient init 2024-09-16T03:05:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient: init 2024-09-16T03:05:03.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:03.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:03.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): picked mon.noname-a con 0x7fc35409e420 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:03.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): picked mon.noname-b con 0x7fc35409e800 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:03.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): picked mon.noname-c con 0x7fc35409ec10 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:03.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3541336a0 con 0x7fc35409e420 2024-09-16T03:05:03.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc354130e40 con 0x7fc35409e800 2024-09-16T03:05:03.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3540a76f0 con 0x7fc35409ec10 2024-09-16T03:05:03.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35c394640 10 monclient(hunting): authenticate will time out at 994.551711s 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35a931640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc35409ec10 0x7fc3541d5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35338/0 (socket says 172.21.15.45:35338) 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35a130640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc35409e420 0x7fc354130020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42640/0 (socket says 172.21.15.45:42640) 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc35a931640 1 -- 172.21.15.45:0/3412862643 learned_addr learned my addr 172.21.15.45:0/3412862643 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc34affd640 10 client.?.objecter ms_handle_connect 0x7fc35409ec10 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc34affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc34affd640 10 client.?.objecter ms_handle_connect 0x7fc35409e420 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.898+0000 7fc34affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 client.?.objecter ms_handle_connect 0x7fc35409e800 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1709516368 0 0) 0x7fc3541336a0 con 0x7fc35409e420 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): my global_id is 4371 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc338004ec0 con 0x7fc35409e420 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723031107 0 0) 0x7fc3540a76f0 con 0x7fc35409ec10 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): my global_id is 4397 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3541336a0 con 0x7fc35409ec10 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1103247410 0 0) 0x7fc354130e40 con 0x7fc35409e800 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): my global_id is 4390 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3540a76f0 con 0x7fc35409e800 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4192698103 0 0) 0x7fc338004ec0 con 0x7fc35409e420 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc354130e40 con 0x7fc35409e420 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc344001860 con 0x7fc35409e420 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): dump: 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3196131697 0 0) 0x7fc3541336a0 con 0x7fc35409ec10 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc338004ec0 con 0x7fc35409ec10 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc34c001f10 con 0x7fc35409ec10 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient(hunting): dump: 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.919 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2193786118 0 0) 0x7fc354130e40 con 0x7fc35409e420 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 >> v1:172.21.15.45:6790/0 conn(0x7fc35409ec10 legacy=0x7fc3541d5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 >> v1:172.21.15.46:6789/0 conn(0x7fc35409e800 legacy=0x7fc354130730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 monclient: found mon.a 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35409dcd0 con 0x7fc35409e420 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient: _finish_auth 0 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 5 monclient: authenticate success, global_id 4371 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 10 monclient: _renew_subs 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc35409f2e0 con 0x7fc35409e420 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3440047f0 con 0x7fc35409e420 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc344004b40 con 0x7fc35409e420 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 monclient: dump: 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35892d640 4 set_mon_vals no callback set 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 10 monclient: _renew_subs 2024-09-16T03:05:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3541333c0 con 0x7fc35409e420 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 1 librados: init done 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fc344010bb0 con 0x7fc35409e420 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3541344e0 con 0x7fc35409e420 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7fc34404aec0 con 0x7fc35409e420 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 client.4371.objecter ms_dispatch 0x7fc354000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 3 client.4371.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 3 client.4371.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 20 client.4371.objecter dump_active .. 0 homeless 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc35992f640 10 monclient: get_auth_request con 0x7fc33803e220 auth_method 0 2024-09-16T03:05:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.902+0000 7fc34affd640 10 client.4371.objecter ms_handle_connect 0x7fc33803e220 2024-09-16T03:05:03.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.906+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fc344016880 con 0x7fc35409e420 2024-09-16T03:05:03.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.906+0000 7fc34affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:03.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:03.906+0000 7fc34affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:04.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.074+0000 7fc35c394640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:05:04.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.074+0000 7fc35c394640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:05:04.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.074+0000 7fc35c394640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:04.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.074+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc354138880 con 0x7fc35409e420 2024-09-16T03:05:04.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.074+0000 7fc34affd640 1 -- 172.21.15.45:0/3412862643 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10878 (unknown 3413328019 0 1323515444) 0x7fc34401ac00 con 0x7fc35409e420 2024-09-16T03:05:04.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.074+0000 7fc34affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:05:04.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.074+0000 7fc34affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:04.079 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:05:04.079 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":20,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","created":"2024-09-16T03:04:38.421437+0000","modified":"2024-09-16T03:05:03.161096+0000","last_up_change":"2024-09-16T03:04:48.525305+0000","last_in_change":"2024-09-16T03:04:40.127320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:04:49.504411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:04:55.369938+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60a0a607-fe6c-4bd5-b369-e0dd6f6bd551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6800","nonce":3131549395}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6801","nonce":3131549395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6803","nonce":3131549395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6802","nonce":3131549395}]},"public_addr":"172.21.15.45:6800/3131549395","cluster_addr":"172.21.15.45:6801/3131549395","heartbeat_back_addr":"172.21.15.45:6803/3131549395","heartbeat_front_addr":"172.21.15.45:6802/3131549395","state":["exists","up"]},{"osd":1,"uuid":"10adca9c-3542-4901-a439-a1c9e9fcb8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6808","nonce":3228817795}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6809","nonce":3228817795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6811","nonce":3228817795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6810","nonce":3228817795}]},"public_addr":"172.21.15.45:6808/3228817795","cluster_addr":"172.21.15.45:6809/3228817795","heartbeat_back_addr":"172.21.15.45:6811/3228817795","heartbeat_front_addr":"172.21.15.45:6810/3228817795","state":["exists","up"]},{"osd":2,"uuid":"88fe1895-098a-4f13-95b2-4ead1739e71c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6804","nonce":207876738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6805","nonce":207876738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6807","nonce":207876738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6806","nonce":207876738}]},"public_addr":"172.21.15.45:6804/207876738","cluster_addr":"172.21.15.45:6805/207876738","heartbeat_back_addr":"172.21.15.45:6807/207876738","heartbeat_front_addr":"172.21.15.45:6806/207876738","state":["exists","up"]},{"osd":3,"uuid":"b04f94bc-5631-478e-904c-1dfc9a673a7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6808","nonce":3088806475}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6809","nonce":3088806475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6811","nonce":3088806475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6810","nonce":3088806475}]},"public_addr":"172.21.15.46:6808/3088806475","cluster_addr":"172.21.15.46:6809/3088806475","heartbeat_back_addr":"172.21.15.46:6811/3088806475","heartbeat_front_addr":"172.21.15.46:6810/3088806475","state":["exists","up"]},{"osd":4,"uuid":"c2dbca96-afb9-48ed-8a71-8a2bea37d152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6800","nonce":264199318}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":264199318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":264199318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":264199318}]},"public_addr":"172.21.15.46:6800/264199318","cluster_addr":"172.21.15.46:6801/264199318","heartbeat_back_addr":"172.21.15.46:6803/264199318","heartbeat_front_addr":"172.21.15.46:6802/264199318","state":["exists","up"]},{"osd":5,"uuid":"dabb8542-5498-4389-ad9e-0c0a54387723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":3390152579}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6805","nonce":3390152579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6807","nonce":3390152579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6806","nonce":3390152579}]},"public_addr":"172.21.15.46:6804/3390152579","cluster_addr":"172.21.15.46:6805/3390152579","heartbeat_back_addr":"172.21.15.46:6807/3390152579","heartbeat_front_addr":"172.21.15.46:6806/3390152579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.249472+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.780224+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.456157+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.830776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.656408+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.995384+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.074+0000 7fc35c394640 10 librados: watch_flush enter 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 10 librados: watch_flush exit 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 20 client.4371.objecter shutdown clearing up homeless session... 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 10 client.4371.objecter successfully canceled tick 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 >> v1:172.21.15.46:6812/149360280 conn(0x7fc33803e220 legacy=0x7fc3380406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 10 monclient: shutdown 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 >> v1:172.21.15.45:6789/0 conn(0x7fc35409e420 legacy=0x7fc354130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 shutdown_connections 2024-09-16T03:05:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 >> 172.21.15.45:0/3412862643 conn(0x7fc3540a4ff0 msgr2=0x7fc35409f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 shutdown_connections 2024-09-16T03:05:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 1 -- 172.21.15.45:0/3412862643 wait complete. 2024-09-16T03:05:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.078+0000 7fc35c394640 1 librados: shutdown 2024-09-16T03:05:04.096 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T03:04:49.504411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T03:04:55.369938+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T03:05:04.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T03:05:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient: get_monmap_and_config 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient: build_initial_monmap 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient: monmap: 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:04.226114+0000 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:04.226114+0000 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:04.227 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 1 Processor -- start 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 1 -- start start 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient: init 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient(hunting): picked mon.noname-b con 0x7f04f80a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient(hunting): picked mon.noname-a con 0x7f04f80a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient(hunting): picked mon.noname-c con 0x7f04f8073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f809d9e0 con 0x7f04f80a5810 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f809dc20 con 0x7f04f8073d40 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f8073360 con 0x7f04f80a4d20 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f7fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f04f80a4d20 0x7f04f80a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35350/0 (socket says 172.21.15.45:35350) 2024-09-16T03:05:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f7fff640 1 -- 172.21.15.45:0/1100777603 learned_addr learned my addr 172.21.15.45:0/1100777603 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04fe87f640 10 monclient(hunting): authenticate will time out at 994.875603s 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 497125131 0 0) 0x7f04f809d9e0 con 0x7f04f80a5810 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 10 monclient(hunting): my global_id is 4374 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04e0003530 con 0x7f04f80a5810 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765459578 0 0) 0x7f04f8073360 con 0x7f04f80a4d20 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.222+0000 7f04f6ffd640 10 monclient(hunting): my global_id is 4400 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04f809d9e0 con 0x7f04f80a4d20 2024-09-16T03:05:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 50149293 0 0) 0x7f04e0003530 con 0x7f04f80a5810 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 >> v1:172.21.15.45:6790/0 conn(0x7f04f80a4d20 legacy=0x7f04f80a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 >> v1:172.21.15.46:6789/0 conn(0x7f04f8073d40 legacy=0x7f04f8140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 1 monclient: found mon.noname-a 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f809cf30 con 0x7f04f80a5810 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f04ec0016a0 con 0x7f04f80a5810 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 5 monclient: authenticate success, global_id 4374 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: dump: 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f04ec0018c0 con 0x7f04f80a5810 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 1 -- 172.21.15.45:0/1100777603 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f04ec001d50 con 0x7f04f80a5810 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 4 set_mon_vals no callback set 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f6ffd640 10 monclient: dump: 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: shutdown 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- 172.21.15.45:0/1100777603 >> v1:172.21.15.45:6789/0 conn(0x7f04f80a5810 legacy=0x7f04f813cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- 172.21.15.45:0/1100777603 shutdown_connections 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- 172.21.15.45:0/1100777603 >> 172.21.15.45:0/1100777603 conn(0x7f04f80aa720 msgr2=0x7f04f80aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- 172.21.15.45:0/1100777603 shutdown_connections 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- 172.21.15.45:0/1100777603 wait complete. 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: build_initial_monmap 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: monmap: 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 librados: starting msgr at 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 librados: starting objecter 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 Processor -- start 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- start start 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 librados: setting wanted keys 2024-09-16T03:05:04.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 librados: calling monclient init 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: init 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient(hunting): picked mon.noname-b con 0x7f04f8073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient(hunting): picked mon.noname-c con 0x7f04f80a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient(hunting): picked mon.noname-a con 0x7f04f80a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f813f180 con 0x7f04f80a5810 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f80a1d20 con 0x7f04f8073d40 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f80a1510 con 0x7f04f80a4d20 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient(hunting): authenticate will time out at 994.878526s 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fce1c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f04f80a5810 0x7f04f81d58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42652/0 (socket says 172.21.15.45:42652) 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fce1c640 1 -- 172.21.15.45:0/3512024593 learned_addr learned my addr 172.21.15.45:0/3512024593 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:04.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 client.?.objecter ms_handle_connect 0x7f04f80a4d20 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 client.?.objecter ms_handle_connect 0x7f04f80a5810 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3096897026 0 0) 0x7f04f80a1510 con 0x7f04f80a4d20 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): my global_id is 4403 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04e4004ef0 con 0x7f04f80a4d20 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 client.?.objecter ms_handle_connect 0x7f04f8073d40 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1476066454 0 0) 0x7f04f813f180 con 0x7f04f80a5810 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): my global_id is 4377 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04f80a1510 con 0x7f04f80a5810 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1593172070 0 0) 0x7f04e4004ef0 con 0x7f04f80a4d20 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04f813f180 con 0x7f04f80a4d20 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f04ec001c10 con 0x7f04f80a4d20 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): dump: 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.243 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369506906 0 0) 0x7f04f80a1d20 con 0x7f04f8073d40 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): my global_id is 4396 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04e4004ef0 con 0x7f04f8073d40 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2634776586 0 0) 0x7f04f80a1510 con 0x7f04f80a5810 2024-09-16T03:05:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04f80a1d20 con 0x7f04f80a5810 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f04f0001c40 con 0x7f04f80a5810 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient(hunting): dump: 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 881686898 0 0) 0x7f04f813f180 con 0x7f04f80a4d20 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 >> v1:172.21.15.46:6789/0 conn(0x7f04f8073d40 legacy=0x7f04f81cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 >> v1:172.21.15.45:6789/0 conn(0x7f04f80a5810 legacy=0x7f04f81d58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 monclient: found mon.c 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f813e1e0 con 0x7f04f80a4d20 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient: _finish_auth 0 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 5 monclient: authenticate success, global_id 4403 2024-09-16T03:05:04.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: _renew_subs 2024-09-16T03:05:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f04f8140260 con 0x7f04f80a4d20 2024-09-16T03:05:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: _renew_subs 2024-09-16T03:05:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f04f8138b10 con 0x7f04f80a4d20 2024-09-16T03:05:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04fe87f640 1 librados: init done 2024-09-16T03:05:04.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f04ec0039d0 con 0x7f04f80a4d20 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f04ec003b80 con 0x7f04f80a4d20 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f67fc640 4 set_mon_vals no callback set 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.226+0000 7f04f4ff9640 10 monclient: dump: 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.248 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04fe87f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04fe87f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04fe87f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d8005090 con 0x7f04f80a4d20 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f04ec00fbf0 con 0x7f04f80a4d20 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f04ec049e50 con 0x7f04f80a4d20 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04f4ff9640 10 client.4403.objecter ms_dispatch 0x7f04f8000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04f4ff9640 3 client.4403.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04f4ff9640 3 client.4403.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04f4ff9640 20 client.4403.objecter dump_active .. 0 homeless 2024-09-16T03:05:04.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04f7fff640 10 monclient: get_auth_request con 0x7f04e403e2e0 auth_method 0 2024-09-16T03:05:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.230+0000 7f04f4ff9640 10 client.4403.objecter ms_handle_connect 0x7f04e403e2e0 2024-09-16T03:05:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.234+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f04ec0158d0 con 0x7f04f80a4d20 2024-09-16T03:05:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.234+0000 7f04f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:04.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.234+0000 7f04f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:04.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.402+0000 7f04fe87f640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T03:05:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.402+0000 7f04fe87f640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T03:05:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.402+0000 7f04fe87f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.402+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f04d8005c50 con 0x7f04f80a4d20 2024-09-16T03:05:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.402+0000 7f04f4ff9640 1 -- 172.21.15.45:0/3512024593 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (unknown 3471955896 0 2170607528) 0x7f04ec0151d0 con 0x7f04f80a4d20 2024-09-16T03:05:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.402+0000 7f04f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T03:05:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.402+0000 7f04f4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:04.409 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-09-16T03:05:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 10 librados: watch_flush enter 2024-09-16T03:05:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 10 librados: watch_flush exit 2024-09-16T03:05:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 20 client.4403.objecter shutdown clearing up homeless session... 2024-09-16T03:05:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 10 client.4403.objecter successfully canceled tick 2024-09-16T03:05:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 >> v1:172.21.15.46:6812/149360280 conn(0x7f04e403e2e0 legacy=0x7f04e40407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 10 monclient: shutdown 2024-09-16T03:05:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 >> v1:172.21.15.45:6790/0 conn(0x7f04f80a4d20 legacy=0x7f04f81d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 shutdown_connections 2024-09-16T03:05:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 >> 172.21.15.45:0/3512024593 conn(0x7f04f80aa720 msgr2=0x7f04f80a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 shutdown_connections 2024-09-16T03:05:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 1 -- 172.21.15.45:0/3512024593 wait complete. 2024-09-16T03:05:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.406+0000 7f04fe87f640 1 librados: shutdown 2024-09-16T03:05:04.426 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T03:05:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.534+0000 7f09e2d91640 10 monclient: get_monmap_and_config 2024-09-16T03:05:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.534+0000 7f09e2d91640 10 monclient: build_initial_monmap 2024-09-16T03:05:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.534+0000 7f09e2d91640 10 monclient: monmap: 2024-09-16T03:05:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:04.539374+0000 2024-09-16T03:05:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:04.539374+0000 2024-09-16T03:05:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:04.542 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.534+0000 7f09e2d91640 1 Processor -- start 2024-09-16T03:05:04.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- start start 2024-09-16T03:05:04.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: init 2024-09-16T03:05:04.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:04.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): picked mon.noname-a con 0x7f09dc09b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): picked mon.noname-c con 0x7f09dc09b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): picked mon.noname-b con 0x7f09dc13b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09dc09cc20 con 0x7f09dc09b8e0 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09dc09d820 con 0x7f09dc09b250 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09dc09b6a0 con 0x7f09dc13b000 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): authenticate will time out at 995.188802s 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e0b2d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f09dc09b8e0 0x7f09dc09bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42666/0 (socket says 172.21.15.45:42666) 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e0b2d640 1 -- 172.21.15.45:0/306721098 learned_addr learned my addr 172.21.15.45:0/306721098 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592003872 0 0) 0x7f09dc09b6a0 con 0x7f09dc13b000 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient(hunting): my global_id is 4406 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09c4003530 con 0x7f09dc13b000 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 281823826 0 0) 0x7f09dc09cc20 con 0x7f09dc09b8e0 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient(hunting): my global_id is 4380 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09dc09b6a0 con 0x7f09dc09b8e0 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1698866380 0 0) 0x7f09c4003530 con 0x7f09dc13b000 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 >> v1:172.21.15.46:6789/0 conn(0x7f09dc09b250 legacy=0x7f09dc1398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 >> v1:172.21.15.45:6789/0 conn(0x7f09dc09b8e0 legacy=0x7f09dc09bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:05:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 monclient: found mon.noname-b 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc0a21f0 con 0x7f09dc13b000 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: _finish_auth 0 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f09d4001a20 con 0x7f09dc13b000 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 5 monclient: authenticate success, global_id 4406 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: dump: 2024-09-16T03:05:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f09d4001bc0 con 0x7f09dc13b000 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 1 -- 172.21.15.45:0/306721098 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f09d4001fd0 con 0x7f09dc13b000 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 4 set_mon_vals no callback set 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09db7fe640 10 monclient: dump: 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: shutdown 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- 172.21.15.45:0/306721098 >> v1:172.21.15.45:6790/0 conn(0x7f09dc13b000 legacy=0x7f09dc13d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- 172.21.15.45:0/306721098 shutdown_connections 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- 172.21.15.45:0/306721098 >> 172.21.15.45:0/306721098 conn(0x7f09dc0a2bd0 msgr2=0x7f09dc0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- 172.21.15.45:0/306721098 shutdown_connections 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- 172.21.15.45:0/306721098 wait complete. 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: build_initial_monmap 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: monmap: 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 librados: starting msgr at 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 librados: starting objecter 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 Processor -- start 2024-09-16T03:05:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- start start 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 librados: setting wanted keys 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 librados: calling monclient init 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: init 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): picked mon.noname-a con 0x7f09dc09b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): picked mon.noname-b con 0x7f09dc09b8e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): picked mon.noname-c con 0x7f09dc13b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09dc13ddd0 con 0x7f09dc09b250 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09dc0948c0 con 0x7f09dc09b8e0 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09dc09d2c0 con 0x7f09dc13b000 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e2d91640 10 monclient(hunting): authenticate will time out at 995.191854s 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e0b2d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f09dc09b250 0x7f09dc1ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42680/0 (socket says 172.21.15.45:42680) 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.538+0000 7f09e0b2d640 1 -- 172.21.15.45:0/2840474559 learned_addr learned my addr 172.21.15.45:0/2840474559 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 client.?.objecter ms_handle_connect 0x7f09dc09b250 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 client.?.objecter ms_handle_connect 0x7f09dc13b000 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110118288 0 0) 0x7f09cc001900 con 0x7f09dc09b250 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): my global_id is 4383 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09c8004ec0 con 0x7f09dc09b250 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2731819581 0 0) 0x7f09dc09d2c0 con 0x7f09dc13b000 2024-09-16T03:05:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): my global_id is 4409 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09cc001900 con 0x7f09dc13b000 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 client.?.objecter ms_handle_connect 0x7f09dc09b8e0 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1798119846 0 0) 0x7f09c8004ec0 con 0x7f09dc09b250 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09dc09d2c0 con 0x7f09dc09b250 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f09cc001c10 con 0x7f09dc09b250 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): dump: 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2452452973 0 0) 0x7f09cc001900 con 0x7f09dc13b000 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09c8004ec0 con 0x7f09dc13b000 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f09d4001d70 con 0x7f09dc13b000 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:04.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): dump: 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 501441688 0 0) 0x7f09dc0948c0 con 0x7f09dc09b8e0 2024-09-16T03:05:04.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient(hunting): my global_id is 4402 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09cc001900 con 0x7f09dc09b8e0 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 250988985 0 0) 0x7f09dc09d2c0 con 0x7f09dc09b250 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 >> v1:172.21.15.45:6790/0 conn(0x7f09dc13b000 legacy=0x7f09dc1d5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 >> v1:172.21.15.46:6789/0 conn(0x7f09dc09b8e0 legacy=0x7f09dc1d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient: _finish_hunting 0 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 monclient: found mon.a 2024-09-16T03:05:04.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc13cd50 con 0x7f09dc09b250 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient: _finish_auth 0 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient: _check_auth_tickets 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 5 monclient: authenticate success, global_id 4383 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 10 monclient: _renew_subs 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09dc0945e0 con 0x7f09dc09b250 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 10 monclient: _renew_subs 2024-09-16T03:05:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f09dc13c3f0 con 0x7f09dc09b250 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 1 librados: init done 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f09cc004aa0 con 0x7f09dc09b250 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f09cc004df0 con 0x7f09dc09b250 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09daffd640 4 set_mon_vals no callback set 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 monclient: dump: 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09bc005090 con 0x7f09dc09b250 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f09cc010e60 con 0x7f09dc09b250 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f09cc04b260 con 0x7f09dc09b250 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 client.4383.objecter ms_dispatch 0x7f09dc000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 3 client.4383.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 3 client.4383.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 20 client.4383.objecter dump_active .. 0 homeless 2024-09-16T03:05:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09dbfff640 10 monclient: get_auth_request con 0x7f09c803e400 auth_method 0 2024-09-16T03:05:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.542+0000 7f09d97fa640 10 client.4383.objecter ms_handle_connect 0x7f09c803e400 2024-09-16T03:05:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.546+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f09dc13ddd0 con 0x7f09dc09b250 2024-09-16T03:05:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.546+0000 7f09d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.546+0000 7f09d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:04.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.718+0000 7f09e2d91640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T03:05:04.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.718+0000 7f09e2d91640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T03:05:04.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.718+0000 7f09e2d91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:04.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.718+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f09bc005c50 con 0x7f09dc09b250 2024-09-16T03:05:04.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.718+0000 7f09d97fa640 1 -- 172.21.15.45:0/2840474559 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (unknown 3412311620 0 265718887) 0x7f09cc016dc0 con 0x7f09dc09b250 2024-09-16T03:05:04.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.718+0000 7f09d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T03:05:04.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.718+0000 7f09d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:04.725 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 8 2024-09-16T03:05:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 10 librados: watch_flush enter 2024-09-16T03:05:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 10 librados: watch_flush exit 2024-09-16T03:05:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 20 client.4383.objecter shutdown clearing up homeless session... 2024-09-16T03:05:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 10 client.4383.objecter successfully canceled tick 2024-09-16T03:05:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 >> v1:172.21.15.46:6812/149360280 conn(0x7f09c803e400 legacy=0x7f09c80408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 10 monclient: shutdown 2024-09-16T03:05:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 >> v1:172.21.15.45:6789/0 conn(0x7f09dc09b250 legacy=0x7f09dc1ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 shutdown_connections 2024-09-16T03:05:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 >> 172.21.15.45:0/2840474559 conn(0x7f09dc0a2bd0 msgr2=0x7f09dc09e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 shutdown_connections 2024-09-16T03:05:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 1 -- 172.21.15.45:0/2840474559 wait complete. 2024-09-16T03:05:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.722+0000 7f09e2d91640 1 librados: shutdown 2024-09-16T03:05:04.743 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T03:05:04.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:05:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.858+0000 7f60fffe1640 10 monclient: get_monmap_and_config 2024-09-16T03:05:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.858+0000 7f60fffe1640 10 monclient: build_initial_monmap 2024-09-16T03:05:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.858+0000 7f60fffe1640 10 monclient: monmap: 2024-09-16T03:05:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:04.863394+0000 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:04.863394+0000 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.858+0000 7f60fffe1640 1 Processor -- start 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- start start 2024-09-16T03:05:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient: init 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient(hunting): picked mon.noname-b con 0x7f60f809b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient(hunting): picked mon.noname-c con 0x7f60f809b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient(hunting): picked mon.noname-a con 0x7f60f813b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f809cc20 con 0x7f60f813b000 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f809d820 con 0x7f60f809b250 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f809b6a0 con 0x7f60f809b8e0 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient(hunting): authenticate will time out at 995.512848s 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fe57e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f60f813b000 0x7f60f813d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42696/0 (socket says 172.21.15.45:42696) 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fdd7d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f60f809b8e0 0x7f60f809bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35390/0 (socket says 172.21.15.45:35390) 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fe57e640 1 -- 172.21.15.45:0/3444169201 learned_addr learned my addr 172.21.15.45:0/3444169201 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2136573964 0 0) 0x7f60f809cc20 con 0x7f60f813b000 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): my global_id is 4386 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60e4003530 con 0x7f60f813b000 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2893227376 0 0) 0x7f60f809b6a0 con 0x7f60f809b8e0 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): my global_id is 4412 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60f809cc20 con 0x7f60f809b8e0 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1469765896 0 0) 0x7f60f809d820 con 0x7f60f809b250 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient(hunting): my global_id is 4405 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60f809b6a0 con 0x7f60f809b250 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1916829860 0 0) 0x7f60f809cc20 con 0x7f60f809b8e0 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 >> v1:172.21.15.46:6789/0 conn(0x7f60f809b250 legacy=0x7f60f81398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 >> v1:172.21.15.45:6789/0 conn(0x7f60f813b000 legacy=0x7f60f813d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: _finish_hunting 0 2024-09-16T03:05:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 monclient: found mon.noname-b 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f80a21f0 con 0x7f60f809b8e0 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: _finish_auth 0 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: _check_auth_tickets 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60e8001600 con 0x7f60f809b8e0 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: dump: 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.877 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 5 monclient: authenticate success, global_id 4412 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60e80017a0 con 0x7f60f809b8e0 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 1 -- 172.21.15.45:0/3444169201 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60e8001c30 con 0x7f60f809b8e0 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:04.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 4 set_mon_vals no callback set 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fcd7b640 10 monclient: dump: 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient: shutdown 2024-09-16T03:05:04.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- 172.21.15.45:0/3444169201 >> v1:172.21.15.45:6790/0 conn(0x7f60f809b8e0 legacy=0x7f60f809bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- 172.21.15.45:0/3444169201 shutdown_connections 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- 172.21.15.45:0/3444169201 >> 172.21.15.45:0/3444169201 conn(0x7f60f80a2bd0 msgr2=0x7f60f80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- 172.21.15.45:0/3444169201 shutdown_connections 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- 172.21.15.45:0/3444169201 wait complete. 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient: build_initial_monmap 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient: monmap: 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 librados: starting msgr at 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 librados: starting objecter 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 Processor -- start 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 -- start start 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 librados: setting wanted keys 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 1 librados: calling monclient init 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient: init 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.862+0000 7f60fffe1640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient(hunting): picked mon.noname-b con 0x7f60f809b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient(hunting): picked mon.noname-c con 0x7f60f809b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient(hunting): picked mon.noname-a con 0x7f60f813b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f8090540 con 0x7f60f813b000 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f809d2c0 con 0x7f60f809b250 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f8091d40 con 0x7f60f809b8e0 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient(hunting): authenticate will time out at 995.516119s 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fd57c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f60f809b8e0 0x7f60f81cfd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35398/0 (socket says 172.21.15.45:35398) 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fd57c640 1 -- 172.21.15.45:0/2155303339 learned_addr learned my addr 172.21.15.45:0/2155303339 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fe57e640 1 --1- 172.21.15.45:0/2155303339 >> v1:172.21.15.45:6789/0 conn(0x7f60f813b000 0x7f60f81d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42706/0 (socket says 172.21.15.45:42706) 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 client.?.objecter ms_handle_connect 0x7f60f813b000 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:04.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 client.?.objecter ms_handle_connect 0x7f60f809b8e0 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3926198472 0 0) 0x7f60f8091d40 con 0x7f60f809b8e0 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): my global_id is 4415 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60dc004ef0 con 0x7f60f809b8e0 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 client.?.objecter ms_handle_connect 0x7f60f809b250 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464297462 0 0) 0x7f60f8090540 con 0x7f60f813b000 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): my global_id is 4389 2024-09-16T03:05:04.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60f8091d40 con 0x7f60f813b000 2024-09-16T03:05:04.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757689561 0 0) 0x7f60f809d2c0 con 0x7f60f809b250 2024-09-16T03:05:04.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:04.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:04.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:04.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): my global_id is 4408 2024-09-16T03:05:04.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60f8090540 con 0x7f60f809b250 2024-09-16T03:05:04.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 535060056 0 0) 0x7f60dc004ef0 con 0x7f60f809b8e0 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60f809d2c0 con 0x7f60f809b8e0 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 916833912 0 0) 0x7f60f8091d40 con 0x7f60f813b000 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60dc004ef0 con 0x7f60f813b000 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60f40019a0 con 0x7f60f809b8e0 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): dump: 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.885 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.886 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.886 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.886 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.886 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.886 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:04.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60f00014e0 con 0x7f60f813b000 2024-09-16T03:05:04.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:04.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient(hunting): dump: 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3097750512 0 0) 0x7f60f809d2c0 con 0x7f60f809b8e0 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 >> v1:172.21.15.46:6789/0 conn(0x7f60f809b250 legacy=0x7f60f81cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 >> v1:172.21.15.45:6789/0 conn(0x7f60f813b000 legacy=0x7f60f81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 monclient: found mon.c 2024-09-16T03:05:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f80928a0 con 0x7f60f809b8e0 2024-09-16T03:05:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 5 monclient: authenticate success, global_id 4415 2024-09-16T03:05:04.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient: _renew_subs 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60f809adf0 con 0x7f60f809b8e0 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient: _renew_subs 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f60f808fa90 con 0x7f60f809b8e0 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 1 librados: init done 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60f4001ce0 con 0x7f60f809b8e0 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60f4003a70 con 0x7f60f809b8e0 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:04.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60effff640 4 set_mon_vals no callback set 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 monclient: dump: 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e0005090 con 0x7f60f809b8e0 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f60f4003c90 con 0x7f60f809b8e0 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f60f404b170 con 0x7f60f809b8e0 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 client.4415.objecter ms_dispatch 0x7f60f8000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 3 client.4415.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 3 client.4415.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 20 client.4415.objecter dump_active .. 0 homeless 2024-09-16T03:05:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60fdd7d640 10 monclient: get_auth_request con 0x7f60dc03e4e0 auth_method 0 2024-09-16T03:05:04.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.866+0000 7f60ee7fc640 10 client.4415.objecter ms_handle_connect 0x7f60dc03e4e0 2024-09-16T03:05:04.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.870+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f60f4016b30 con 0x7f60f809b8e0 2024-09-16T03:05:04.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.870+0000 7f60ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:04.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:04.870+0000 7f60ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:05.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.034+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60e0000ba0 con 0x7f60dc03e4e0 2024-09-16T03:05:05.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.038+0000 7f60ee7fc640 1 -- 172.21.15.45:0/2155303339 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34147 (unknown 2965378022 0 3338192136) 0x7f60e0000ba0 con 0x7f60dc03e4e0 2024-09-16T03:05:05.043 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:05:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-16T03:05:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 10 librados: watch_flush enter 2024-09-16T03:05:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 10 librados: watch_flush exit 2024-09-16T03:05:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 20 client.4415.objecter shutdown clearing up homeless session... 2024-09-16T03:05:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 10 client.4415.objecter successfully canceled tick 2024-09-16T03:05:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 >> v1:172.21.15.46:6812/149360280 conn(0x7f60dc03e4e0 legacy=0x7f60dc0409a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 10 monclient: shutdown 2024-09-16T03:05:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:05.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 >> v1:172.21.15.45:6790/0 conn(0x7f60f809b8e0 legacy=0x7f60f81cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 shutdown_connections 2024-09-16T03:05:05.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 >> 172.21.15.45:0/2155303339 conn(0x7f60f80a2bd0 msgr2=0x7f60f80a4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:05.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 shutdown_connections 2024-09-16T03:05:05.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 1 -- 172.21.15.45:0/2155303339 wait complete. 2024-09-16T03:05:05.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.042+0000 7f60fffe1640 1 librados: shutdown 2024-09-16T03:05:05.059 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T03:05:03.460693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":185008,"kb_used_data":2008,"kb_used_omap":9,"kb_used_meta":182838,"kb_avail":566046032,"statfs":{"total":579820584960,"available":579631136768,"internally_reserved":0,"allocated":2056192,"data_stored":1369846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":187226786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001846"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.132050+0000","last_change":"2024-09-16T03:04:59.132718+0000","last_active":"2024-09-16T03:04:59.132050+0000","last_peered":"2024-09-16T03:04:59.132050+0000","last_clean":"2024-09-16T03:04:59.132050+0000","last_became_active":"2024-09-16T03:04:57.691813+0000","last_became_peered":"2024-09-16T03:04:57.691813+0000","last_unstale":"2024-09-16T03:04:59.132050+0000","last_undegraded":"2024-09-16T03:04:59.132050+0000","last_fullsized":"2024-09-16T03:04:59.132050+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:55.115695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015712199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.142318+0000","last_change":"2024-09-16T03:04:59.142318+0000","last_active":"2024-09-16T03:04:59.142318+0000","last_peered":"2024-09-16T03:04:59.142318+0000","last_clean":"2024-09-16T03:04:59.142318+0000","last_became_active":"2024-09-16T03:04:57.117464+0000","last_became_peered":"2024-09-16T03:04:57.117464+0000","last_unstale":"2024-09-16T03:04:59.142318+0000","last_undegraded":"2024-09-16T03:04:59.142318+0000","last_fullsized":"2024-09-16T03:04:59.142318+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:54:40.258398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012748499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.131459+0000","last_change":"2024-09-16T03:04:59.131904+0000","last_active":"2024-09-16T03:04:59.131459+0000","last_peered":"2024-09-16T03:04:59.131459+0000","last_clean":"2024-09-16T03:04:59.131459+0000","last_became_active":"2024-09-16T03:04:57.113180+0000","last_became_peered":"2024-09-16T03:04:57.113180+0000","last_unstale":"2024-09-16T03:04:59.131459+0000","last_undegraded":"2024-09-16T03:04:59.131459+0000","last_fullsized":"2024-09-16T03:04:59.131459+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:30.278408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087237100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.142273+0000","last_change":"2024-09-16T03:04:59.142273+0000","last_active":"2024-09-16T03:04:59.142273+0000","last_peered":"2024-09-16T03:04:59.142273+0000","last_clean":"2024-09-16T03:04:59.142273+0000","last_became_active":"2024-09-16T03:04:57.117660+0000","last_became_peered":"2024-09-16T03:04:57.117660+0000","last_unstale":"2024-09-16T03:04:59.142273+0000","last_undegraded":"2024-09-16T03:04:59.142273+0000","last_fullsized":"2024-09-16T03:04:59.142273+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:22.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.138496+0000","last_change":"2024-09-16T03:04:59.138496+0000","last_active":"2024-09-16T03:04:59.138496+0000","last_peered":"2024-09-16T03:04:59.138496+0000","last_clean":"2024-09-16T03:04:59.138496+0000","last_became_active":"2024-09-16T03:04:57.119695+0000","last_became_peered":"2024-09-16T03:04:57.119695+0000","last_unstale":"2024-09-16T03:04:59.138496+0000","last_undegraded":"2024-09-16T03:04:59.138496+0000","last_fullsized":"2024-09-16T03:04:59.138496+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:24.631318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024209740000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.132613+0000","last_change":"2024-09-16T03:04:59.133304+0000","last_active":"2024-09-16T03:04:59.132613+0000","last_peered":"2024-09-16T03:04:59.132613+0000","last_clean":"2024-09-16T03:04:59.132613+0000","last_became_active":"2024-09-16T03:04:57.118413+0000","last_became_peered":"2024-09-16T03:04:57.118413+0000","last_unstale":"2024-09-16T03:04:59.132613+0000","last_undegraded":"2024-09-16T03:04:59.132613+0000","last_fullsized":"2024-09-16T03:04:59.132613+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:24.300841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014587720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.131619+0000","last_change":"2024-09-16T03:04:59.132034+0000","last_active":"2024-09-16T03:04:59.131619+0000","last_peered":"2024-09-16T03:04:59.131619+0000","last_clean":"2024-09-16T03:04:59.131619+0000","last_became_active":"2024-09-16T03:04:57.117048+0000","last_became_peered":"2024-09-16T03:04:57.117048+0000","last_unstale":"2024-09-16T03:04:59.131619+0000","last_undegraded":"2024-09-16T03:04:59.131619+0000","last_fullsized":"2024-09-16T03:04:59.131619+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:57.899748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099232000000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.136747+0000","last_change":"2024-09-16T03:04:59.137133+0000","last_active":"2024-09-16T03:04:59.136747+0000","last_peered":"2024-09-16T03:04:59.136747+0000","last_clean":"2024-09-16T03:04:59.136747+0000","last_became_active":"2024-09-16T03:04:57.116849+0000","last_became_peered":"2024-09-16T03:04:57.116849+0000","last_unstale":"2024-09-16T03:04:59.136747+0000","last_undegraded":"2024-09-16T03:04:59.136747+0000","last_fullsized":"2024-09-16T03:04:59.136747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:51.385856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00103192,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.130859+0000","last_change":"2024-09-16T03:04:51.319467+0000","last_active":"2024-09-16T03:04:59.130859+0000","last_peered":"2024-09-16T03:04:59.130859+0000","last_clean":"2024-09-16T03:04:59.130859+0000","last_became_active":"2024-09-16T03:04:51.317160+0000","last_became_peered":"2024-09-16T03:04:51.317160+0000","last_unstale":"2024-09-16T03:04:59.130859+0000","last_undegraded":"2024-09-16T03:04:59.130859+0000","last_fullsized":"2024-09-16T03:04:59.130859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_clean_scrub_stamp":"2024-09-16T03:04:49.540253+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:15.016218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:05:05.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:05:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.162+0000 7f094af52640 10 monclient: get_monmap_and_config 2024-09-16T03:05:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.162+0000 7f094af52640 10 monclient: build_initial_monmap 2024-09-16T03:05:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.162+0000 7f094af52640 10 monclient: monmap: 2024-09-16T03:05:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:05.167746+0000 2024-09-16T03:05:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:05.167746+0000 2024-09-16T03:05:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:05.170 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 Processor -- start 2024-09-16T03:05:05.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 -- start start 2024-09-16T03:05:05.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient: init 2024-09-16T03:05:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient(hunting): picked mon.noname-c con 0x7f0944068960 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient(hunting): picked mon.noname-a con 0x7f094409dc60 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient(hunting): picked mon.noname-b con 0x7f0944069450 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09440a2680 con 0x7f094409dc60 2024-09-16T03:05:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09440a1d70 con 0x7f0944068960 2024-09-16T03:05:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09441413d0 con 0x7f0944069450 2024-09-16T03:05:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient(hunting): authenticate will time out at 995.816999s 2024-09-16T03:05:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f094409dc60 0x7f0944068d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42714/0 (socket says 172.21.15.45:42714) 2024-09-16T03:05:05.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f09494ef640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0944069450 0x7f094413dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35406/0 (socket says 172.21.15.45:35406) 2024-09-16T03:05:05.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093bfff640 1 -- 172.21.15.45:0/691035355 learned_addr learned my addr 172.21.15.45:0/691035355 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:05.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 492351106 0 0) 0x7f09441413d0 con 0x7f0944069450 2024-09-16T03:05:05.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient(hunting): my global_id is 4418 2024-09-16T03:05:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f092c003530 con 0x7f0944069450 2024-09-16T03:05:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352287134 0 0) 0x7f09440a2680 con 0x7f094409dc60 2024-09-16T03:05:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient(hunting): my global_id is 4392 2024-09-16T03:05:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09441413d0 con 0x7f094409dc60 2024-09-16T03:05:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3714619350 0 0) 0x7f09441413d0 con 0x7f094409dc60 2024-09-16T03:05:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 >> v1:172.21.15.45:6790/0 conn(0x7f0944069450 legacy=0x7f094413dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 >> v1:172.21.15.46:6789/0 conn(0x7f0944068960 legacy=0x7f09440a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:05:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:05:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:05.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09441425b0 con 0x7f094409dc60 2024-09-16T03:05:05.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:05:05.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:05:05.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0934001680 con 0x7f094409dc60 2024-09-16T03:05:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 5 monclient: authenticate success, global_id 4392 2024-09-16T03:05:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:05.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:05.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: dump: 2024-09-16T03:05:05.192 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.192 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.192 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.192 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.194 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f09340018a0 con 0x7f094409dc60 2024-09-16T03:05:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 1 -- 172.21.15.45:0/691035355 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0934001c30 con 0x7f094409dc60 2024-09-16T03:05:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f093b7fe640 10 monclient: dump: 2024-09-16T03:05:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 4 set_mon_vals no callback set 2024-09-16T03:05:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient: shutdown 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 -- 172.21.15.45:0/691035355 >> v1:172.21.15.45:6789/0 conn(0x7f094409dc60 legacy=0x7f0944068d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 -- 172.21.15.45:0/691035355 shutdown_connections 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 -- 172.21.15.45:0/691035355 >> 172.21.15.45:0/691035355 conn(0x7f09440a3d20 msgr2=0x7f09440a4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 -- 172.21.15.45:0/691035355 shutdown_connections 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 -- 172.21.15.45:0/691035355 wait complete. 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient: build_initial_monmap 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 10 monclient: monmap: 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 librados: starting msgr at 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.166+0000 7f094af52640 1 librados: starting objecter 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 Processor -- start 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 -- start start 2024-09-16T03:05:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 librados: setting wanted keys 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 librados: calling monclient init 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient: init 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient(hunting): picked mon.noname-b con 0x7f0944068960 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient(hunting): picked mon.noname-c con 0x7f0944069450 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient(hunting): picked mon.noname-a con 0x7f094409dc60 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f094413eb40 con 0x7f094409dc60 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0944141370 con 0x7f0944068960 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f094409e240 con 0x7f0944069450 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient(hunting): authenticate will time out at 995.820898s 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f093bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0944069450 0x7f09441cf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35422/0 (socket says 172.21.15.45:35422) 2024-09-16T03:05:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f093bfff640 1 -- 172.21.15.45:0/1136667903 learned_addr learned my addr 172.21.15.45:0/1136667903 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 client.?.objecter ms_handle_connect 0x7f0944069450 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 client.?.objecter ms_handle_connect 0x7f094409dc60 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1010309092 0 0) 0x7f094409e240 con 0x7f0944069450 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): my global_id is 4421 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0930004ec0 con 0x7f0944069450 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3080834357 0 0) 0x7f094413eb40 con 0x7f094409dc60 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): my global_id is 4395 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f094409e240 con 0x7f094409dc60 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 client.?.objecter ms_handle_connect 0x7f0944068960 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 193831583 0 0) 0x7f094409e240 con 0x7f094409dc60 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f094413eb40 con 0x7f094409dc60 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f093c001ce0 con 0x7f094409dc60 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): dump: 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3762805486 0 0) 0x7f0930004ec0 con 0x7f0944069450 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f094409e240 con 0x7f0944069450 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0934001bf0 con 0x7f0944069450 2024-09-16T03:05:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient(hunting): dump: 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 153174863 0 0) 0x7f094413eb40 con 0x7f094409dc60 2024-09-16T03:05:05.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 >> v1:172.21.15.45:6790/0 conn(0x7f0944069450 legacy=0x7f09441cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 >> v1:172.21.15.46:6789/0 conn(0x7f0944068960 legacy=0x7f0944140070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient: _finish_hunting 0 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 monclient: found mon.a 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094413c7e0 con 0x7f094409dc60 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient: _finish_auth 0 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient: _check_auth_tickets 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 5 monclient: authenticate success, global_id 4395 2024-09-16T03:05:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient: _renew_subs 2024-09-16T03:05:05.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f094413e860 con 0x7f094409dc60 2024-09-16T03:05:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient: _renew_subs 2024-09-16T03:05:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f09441c76f0 con 0x7f094409dc60 2024-09-16T03:05:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 librados: init done 2024-09-16T03:05:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f093c004b70 con 0x7f094409dc60 2024-09-16T03:05:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f093c004f00 con 0x7f094409dc60 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f093affd640 4 set_mon_vals no callback set 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 monclient: dump: 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0924005090 con 0x7f094409dc60 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f093c0050a0 con 0x7f094409dc60 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f093c04b2b0 con 0x7f094409dc60 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 client.4395.objecter ms_dispatch 0x7f0944000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 3 client.4395.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 3 client.4395.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 20 client.4395.objecter dump_active .. 0 homeless 2024-09-16T03:05:05.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f0948cee640 10 monclient: get_auth_request con 0x7f093003e130 auth_method 0 2024-09-16T03:05:05.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.170+0000 7f09397fa640 10 client.4395.objecter ms_handle_connect 0x7f093003e130 2024-09-16T03:05:05.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.174+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f093c016d70 con 0x7f094409dc60 2024-09-16T03:05:05.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.174+0000 7f09397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:05.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.174+0000 7f09397fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:05.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.338+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0924000ba0 con 0x7f093003e130 2024-09-16T03:05:05.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.342+0000 7f09397fa640 1 -- 172.21.15.45:0/1136667903 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34147 (unknown 2965378022 0 3338192136) 0x7f0924000ba0 con 0x7f093003e130 2024-09-16T03:05:05.347 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:05:05.347 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.342+0000 7f094af52640 10 librados: watch_flush enter 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.342+0000 7f094af52640 10 librados: watch_flush exit 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.342+0000 7f094af52640 20 client.4395.objecter shutdown clearing up homeless session... 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.342+0000 7f094af52640 10 client.4395.objecter successfully canceled tick 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.346+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 >> v1:172.21.15.46:6812/149360280 conn(0x7f093003e130 legacy=0x7f09300405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.346+0000 7f094af52640 10 monclient: shutdown 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.346+0000 7f094af52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.346+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 >> v1:172.21.15.45:6789/0 conn(0x7f094409dc60 legacy=0x7f09441d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.346+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 shutdown_connections 2024-09-16T03:05:05.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.346+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 >> 172.21.15.45:0/1136667903 conn(0x7f09440a3d20 msgr2=0x7f0944074400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:05.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.346+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 shutdown_connections 2024-09-16T03:05:05.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.346+0000 7f094af52640 1 -- 172.21.15.45:0/1136667903 wait complete. 2024-09-16T03:05:05.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.346+0000 7f094af52640 1 librados: shutdown 2024-09-16T03:05:05.365 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T03:05:03.460693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":185008,"kb_used_data":2008,"kb_used_omap":9,"kb_used_meta":182838,"kb_avail":566046032,"statfs":{"total":579820584960,"available":579631136768,"internally_reserved":0,"allocated":2056192,"data_stored":1369846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":187226786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001846"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.132050+0000","last_change":"2024-09-16T03:04:59.132718+0000","last_active":"2024-09-16T03:04:59.132050+0000","last_peered":"2024-09-16T03:04:59.132050+0000","last_clean":"2024-09-16T03:04:59.132050+0000","last_became_active":"2024-09-16T03:04:57.691813+0000","last_became_peered":"2024-09-16T03:04:57.691813+0000","last_unstale":"2024-09-16T03:04:59.132050+0000","last_undegraded":"2024-09-16T03:04:59.132050+0000","last_fullsized":"2024-09-16T03:04:59.132050+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:55.115695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015712199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.142318+0000","last_change":"2024-09-16T03:04:59.142318+0000","last_active":"2024-09-16T03:04:59.142318+0000","last_peered":"2024-09-16T03:04:59.142318+0000","last_clean":"2024-09-16T03:04:59.142318+0000","last_became_active":"2024-09-16T03:04:57.117464+0000","last_became_peered":"2024-09-16T03:04:57.117464+0000","last_unstale":"2024-09-16T03:04:59.142318+0000","last_undegraded":"2024-09-16T03:04:59.142318+0000","last_fullsized":"2024-09-16T03:04:59.142318+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:54:40.258398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012748499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.131459+0000","last_change":"2024-09-16T03:04:59.131904+0000","last_active":"2024-09-16T03:04:59.131459+0000","last_peered":"2024-09-16T03:04:59.131459+0000","last_clean":"2024-09-16T03:04:59.131459+0000","last_became_active":"2024-09-16T03:04:57.113180+0000","last_became_peered":"2024-09-16T03:04:57.113180+0000","last_unstale":"2024-09-16T03:04:59.131459+0000","last_undegraded":"2024-09-16T03:04:59.131459+0000","last_fullsized":"2024-09-16T03:04:59.131459+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:30.278408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087237100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.142273+0000","last_change":"2024-09-16T03:04:59.142273+0000","last_active":"2024-09-16T03:04:59.142273+0000","last_peered":"2024-09-16T03:04:59.142273+0000","last_clean":"2024-09-16T03:04:59.142273+0000","last_became_active":"2024-09-16T03:04:57.117660+0000","last_became_peered":"2024-09-16T03:04:57.117660+0000","last_unstale":"2024-09-16T03:04:59.142273+0000","last_undegraded":"2024-09-16T03:04:59.142273+0000","last_fullsized":"2024-09-16T03:04:59.142273+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:22.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.138496+0000","last_change":"2024-09-16T03:04:59.138496+0000","last_active":"2024-09-16T03:04:59.138496+0000","last_peered":"2024-09-16T03:04:59.138496+0000","last_clean":"2024-09-16T03:04:59.138496+0000","last_became_active":"2024-09-16T03:04:57.119695+0000","last_became_peered":"2024-09-16T03:04:57.119695+0000","last_unstale":"2024-09-16T03:04:59.138496+0000","last_undegraded":"2024-09-16T03:04:59.138496+0000","last_fullsized":"2024-09-16T03:04:59.138496+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:24.631318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024209740000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.132613+0000","last_change":"2024-09-16T03:04:59.133304+0000","last_active":"2024-09-16T03:04:59.132613+0000","last_peered":"2024-09-16T03:04:59.132613+0000","last_clean":"2024-09-16T03:04:59.132613+0000","last_became_active":"2024-09-16T03:04:57.118413+0000","last_became_peered":"2024-09-16T03:04:57.118413+0000","last_unstale":"2024-09-16T03:04:59.132613+0000","last_undegraded":"2024-09-16T03:04:59.132613+0000","last_fullsized":"2024-09-16T03:04:59.132613+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:24.300841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014587720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.131619+0000","last_change":"2024-09-16T03:04:59.132034+0000","last_active":"2024-09-16T03:04:59.131619+0000","last_peered":"2024-09-16T03:04:59.131619+0000","last_clean":"2024-09-16T03:04:59.131619+0000","last_became_active":"2024-09-16T03:04:57.117048+0000","last_became_peered":"2024-09-16T03:04:57.117048+0000","last_unstale":"2024-09-16T03:04:59.131619+0000","last_undegraded":"2024-09-16T03:04:59.131619+0000","last_fullsized":"2024-09-16T03:04:59.131619+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:57.899748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099232000000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.136747+0000","last_change":"2024-09-16T03:04:59.137133+0000","last_active":"2024-09-16T03:04:59.136747+0000","last_peered":"2024-09-16T03:04:59.136747+0000","last_clean":"2024-09-16T03:04:59.136747+0000","last_became_active":"2024-09-16T03:04:57.116849+0000","last_became_peered":"2024-09-16T03:04:57.116849+0000","last_unstale":"2024-09-16T03:04:59.136747+0000","last_undegraded":"2024-09-16T03:04:59.136747+0000","last_fullsized":"2024-09-16T03:04:59.136747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:51.385856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00103192,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:04:59.130859+0000","last_change":"2024-09-16T03:04:51.319467+0000","last_active":"2024-09-16T03:04:59.130859+0000","last_peered":"2024-09-16T03:04:59.130859+0000","last_clean":"2024-09-16T03:04:59.130859+0000","last_became_active":"2024-09-16T03:04:51.317160+0000","last_became_peered":"2024-09-16T03:04:51.317160+0000","last_unstale":"2024-09-16T03:04:59.130859+0000","last_undegraded":"2024-09-16T03:04:59.130859+0000","last_fullsized":"2024-09-16T03:04:59.130859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_clean_scrub_stamp":"2024-09-16T03:04:49.540253+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:15.016218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:05:05.366 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T03:05:05.366 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T03:05:05.366 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T03:05:05.381 INFO:tasks.workunit:Pulling workunits from ref 7498493279982dfe87f00616198a5967475f6169 2024-09-16T03:05:05.381 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T03:05:05.381 DEBUG:teuthology.orchestra.run.smithi046:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T03:05:05.382 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T03:05:05.382 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726455905, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-16T03:05:05.382 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:05:05.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:05:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:05:05.389 INFO:teuthology.orchestra.run.smithi046.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T03:05:05.389 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T03:05:05.438 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T03:05:05.439 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T03:05:05.486 INFO:tasks.workunit:timeout=3h 2024-09-16T03:05:05.487 INFO:tasks.workunit:cleanup=True 2024-09-16T03:05:05.487 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7498493279982dfe87f00616198a5967475f6169 2024-09-16T03:05:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient: get_monmap_and_config 2024-09-16T03:05:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient: build_initial_monmap 2024-09-16T03:05:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient: monmap: 2024-09-16T03:05:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:05.496956+0000 2024-09-16T03:05:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:05.496956+0000 2024-09-16T03:05:05.498 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:05.498 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.498 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:05.498 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:05.498 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:05.498 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 Processor -- start 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 -- start start 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient: init 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient(hunting): picked mon.noname-a con 0x7f7628135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient(hunting): picked mon.noname-b con 0x7f762809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient(hunting): picked mon.noname-c con 0x7f762813c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762809f7e0 con 0x7f7628135950 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762809b160 con 0x7f762813c010 2024-09-16T03:05:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762809b440 con 0x7f762809fc40 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient(hunting): authenticate will time out at 996.146106s 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762e224640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7628135950 0x7f7628135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42742/0 (socket says 172.21.15.45:42742) 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762da23640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f762809fc40 0x7f762813a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35432/0 (socket says 172.21.15.45:35432) 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762e224640 1 -- 172.21.15.45:0/85516219 learned_addr learned my addr 172.21.15.45:0/85516219 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2683167613 0 0) 0x7f762809f7e0 con 0x7f7628135950 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient(hunting): my global_id is 4398 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7614003530 con 0x7f7628135950 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 650749051 0 0) 0x7f762809b440 con 0x7f762809fc40 2024-09-16T03:05:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient(hunting): my global_id is 4424 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f762809f7e0 con 0x7f762809fc40 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3742269123 0 0) 0x7f7614003530 con 0x7f7628135950 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 >> v1:172.21.15.45:6790/0 conn(0x7f762809fc40 legacy=0x7f762813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 >> v1:172.21.15.46:6789/0 conn(0x7f762813c010 legacy=0x7f762813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: _finish_hunting 0 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 monclient: found mon.noname-a 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7628136440 con 0x7f7628135950 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: _finish_auth 0 2024-09-16T03:05:05.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: _check_auth_tickets 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7624001a20 con 0x7f7628135950 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 5 monclient: authenticate success, global_id 4398 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: dump: 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.504 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7624001c40 con 0x7f7628135950 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 1 -- 172.21.15.45:0/85516219 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7624002090 con 0x7f7628135950 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f762d222640 10 monclient: dump: 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.505 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 4 set_mon_vals no callback set 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 10 monclient: shutdown 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:05.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 -- 172.21.15.45:0/85516219 >> v1:172.21.15.45:6789/0 conn(0x7f7628135950 legacy=0x7f7628135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 -- 172.21.15.45:0/85516219 shutdown_connections 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 -- 172.21.15.45:0/85516219 >> 172.21.15.45:0/85516219 conn(0x7f76280a2bd0 msgr2=0x7f76280a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 -- 172.21.15.45:0/85516219 shutdown_connections 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.494+0000 7f7630488640 1 -- 172.21.15.45:0/85516219 wait complete. 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: build_initial_monmap 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: monmap: 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:05.507 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:05.508 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 librados: starting msgr at 2024-09-16T03:05:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 librados: starting objecter 2024-09-16T03:05:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 Processor -- start 2024-09-16T03:05:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 -- start start 2024-09-16T03:05:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 librados: setting wanted keys 2024-09-16T03:05:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 librados: calling monclient init 2024-09-16T03:05:05.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: init 2024-09-16T03:05:05.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:05.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:05.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient(hunting): picked mon.noname-a con 0x7f762809fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:05.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient(hunting): picked mon.noname-c con 0x7f7628135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:05.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient(hunting): picked mon.noname-b con 0x7f762813c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:05.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7628073dd0 con 0x7f762809fc40 2024-09-16T03:05:05.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7628073200 con 0x7f762813c010 2024-09-16T03:05:05.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762813ad20 con 0x7f7628135950 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient(hunting): authenticate will time out at 996.149007s 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f762e224640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f762809fc40 0x7f7628099290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42754/0 (socket says 172.21.15.45:42754) 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f762e224640 1 -- 172.21.15.45:0/3552674125 learned_addr learned my addr 172.21.15.45:0/3552674125 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 client.?.objecter ms_handle_connect 0x7f762809fc40 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 client.?.objecter ms_handle_connect 0x7f7628135950 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272146503 0 0) 0x7f7628073dd0 con 0x7f762809fc40 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): my global_id is 4401 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f760c004ec0 con 0x7f762809fc40 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 262869248 0 0) 0x7f762813ad20 con 0x7f7628135950 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): my global_id is 4427 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7628073dd0 con 0x7f7628135950 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 client.?.objecter ms_handle_connect 0x7f762813c010 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 464291007 0 0) 0x7f760c004ec0 con 0x7f762809fc40 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f762813ad20 con 0x7f762809fc40 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7624001f90 con 0x7f762809fc40 2024-09-16T03:05:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): dump: 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1372649354 0 0) 0x7f7628073200 con 0x7f762813c010 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): my global_id is 4414 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f760c004ec0 con 0x7f762813c010 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 806813347 0 0) 0x7f7628073dd0 con 0x7f7628135950 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7628073200 con 0x7f7628135950 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f76180017a0 con 0x7f7628135950 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient(hunting): dump: 2024-09-16T03:05:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 816181385 0 0) 0x7f762813ad20 con 0x7f762809fc40 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 >> v1:172.21.15.45:6790/0 conn(0x7f7628135950 legacy=0x7f76281d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 >> v1:172.21.15.46:6789/0 conn(0x7f762813c010 legacy=0x7f76281d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 monclient: found mon.a 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7628073af0 con 0x7f762809fc40 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient: _finish_auth 0 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 5 monclient: authenticate success, global_id 4401 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: _renew_subs 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7628071ec0 con 0x7f762809fc40 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: _renew_subs 2024-09-16T03:05:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7628071440 con 0x7f762809fc40 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 librados: init done 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7624001ba0 con 0x7f762809fc40 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f76240051a0 con 0x7f762809fc40 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f762ca21640 4 set_mon_vals no callback set 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 monclient: dump: 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7610005090 con 0x7f762809fc40 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7624011210 con 0x7f762809fc40 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f762404b480 con 0x7f762809fc40 2024-09-16T03:05:05.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 client.4401.objecter ms_dispatch 0x7f7628000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:05.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 3 client.4401.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:05.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 3 client.4401.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:05.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 20 client.4401.objecter dump_active .. 0 homeless 2024-09-16T03:05:05.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f762da23640 10 monclient: get_auth_request con 0x7f760c03e2b0 auth_method 0 2024-09-16T03:05:05.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.498+0000 7f761effd640 10 client.4401.objecter ms_handle_connect 0x7f760c03e2b0 2024-09-16T03:05:05.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.502+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f7624016f00 con 0x7f762809fc40 2024-09-16T03:05:05.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.502+0000 7f761effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:05.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.502+0000 7f761effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:05.536 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T03:05:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.734+0000 7f7630488640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:05:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.734+0000 7f7630488640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:05:05.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.734+0000 7f7630488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.734+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7610005380 con 0x7f762809fc40 2024-09-16T03:05:05.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.734+0000 7f761effd640 1 -- 172.21.15.45:0/3552674125 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 3670256333) 0x7f7624016800 con 0x7f762809fc40 2024-09-16T03:05:05.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.734+0000 7f761effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:05:05.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.734+0000 7f761effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:05.739 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:05:05.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 10 librados: watch_flush enter 2024-09-16T03:05:05.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 10 librados: watch_flush exit 2024-09-16T03:05:05.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 20 client.4401.objecter shutdown clearing up homeless session... 2024-09-16T03:05:05.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 10 client.4401.objecter successfully canceled tick 2024-09-16T03:05:05.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 >> v1:172.21.15.46:6812/149360280 conn(0x7f760c03e2b0 legacy=0x7f760c040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 10 monclient: shutdown 2024-09-16T03:05:05.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:05.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 >> v1:172.21.15.45:6789/0 conn(0x7f762809fc40 legacy=0x7f7628099290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 shutdown_connections 2024-09-16T03:05:05.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 >> 172.21.15.45:0/3552674125 conn(0x7f76280a2bd0 msgr2=0x7f76280a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:05.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 shutdown_connections 2024-09-16T03:05:05.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 1 -- 172.21.15.45:0/3552674125 wait complete. 2024-09-16T03:05:05.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.738+0000 7f7630488640 1 librados: shutdown 2024-09-16T03:05:05.756 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:05:05.756 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:05:05.756 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:05:05.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:05:05.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient: get_monmap_and_config 2024-09-16T03:05:05.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient: build_initial_monmap 2024-09-16T03:05:05.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient: monmap: 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:05.877047+0000 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:05.877047+0000 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:05.878 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 1 Processor -- start 2024-09-16T03:05:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 1 -- start start 2024-09-16T03:05:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient: init 2024-09-16T03:05:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient(hunting): picked mon.noname-c con 0x7f35e8135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient(hunting): picked mon.noname-b con 0x7f35e809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient(hunting): picked mon.noname-a con 0x7f35e813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e809f7e0 con 0x7f35e813c010 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e809b160 con 0x7f35e8135950 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e809b440 con 0x7f35e809fc40 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient(hunting): authenticate will time out at 996.526527s 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35ed215640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f35e809fc40 0x7f35e813a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35454/0 (socket says 172.21.15.45:35454) 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35ee217640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f35e813c010 0x7f35e813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42770/0 (socket says 172.21.15.45:42770) 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35ed215640 1 -- 172.21.15.45:0/1766907580 learned_addr learned my addr 172.21.15.45:0/1766907580 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182757899 0 0) 0x7f35e809f7e0 con 0x7f35e813c010 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient(hunting): my global_id is 4404 2024-09-16T03:05:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35d4003530 con 0x7f35e813c010 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1278908659 0 0) 0x7f35e809b440 con 0x7f35e809fc40 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient(hunting): my global_id is 4430 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35e809f7e0 con 0x7f35e809fc40 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1447241692 0 0) 0x7f35d4003530 con 0x7f35e813c010 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 >> v1:172.21.15.45:6790/0 conn(0x7f35e809fc40 legacy=0x7f35e813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 >> v1:172.21.15.46:6789/0 conn(0x7f35e8135950 legacy=0x7f35e8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: _finish_hunting 0 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 monclient: found mon.noname-a 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e8136440 con 0x7f35e813c010 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: _finish_auth 0 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: _check_auth_tickets 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f35e0001a20 con 0x7f35e813c010 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 5 monclient: authenticate success, global_id 4404 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: dump: 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35e0001c40 con 0x7f35e813c010 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 1 -- 172.21.15.45:0/1766907580 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f35e00020d0 con 0x7f35e813c010 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 4 set_mon_vals no callback set 2024-09-16T03:05:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35eca14640 10 monclient: dump: 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 10 monclient: shutdown 2024-09-16T03:05:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 1 -- 172.21.15.45:0/1766907580 >> v1:172.21.15.45:6789/0 conn(0x7f35e813c010 legacy=0x7f35e813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 1 -- 172.21.15.45:0/1766907580 shutdown_connections 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.874+0000 7f35efc7a640 1 -- 172.21.15.45:0/1766907580 >> 172.21.15.45:0/1766907580 conn(0x7f35e80a2bd0 msgr2=0x7f35e80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 -- 172.21.15.45:0/1766907580 shutdown_connections 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 -- 172.21.15.45:0/1766907580 wait complete. 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient: build_initial_monmap 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient: monmap: 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 librados: starting msgr at 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 librados: starting objecter 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 Processor -- start 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 -- start start 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 librados: setting wanted keys 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 librados: calling monclient init 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient: init 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient(hunting): picked mon.noname-b con 0x7f35e809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient(hunting): picked mon.noname-a con 0x7f35e8135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient(hunting): picked mon.noname-c con 0x7f35e813c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e8070e10 con 0x7f35e8135950 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e8073180 con 0x7f35e809fc40 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e8073460 con 0x7f35e813c010 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient(hunting): authenticate will time out at 996.529378s 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35ed215640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f35e8135950 0x7f35e8070700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42782/0 (socket says 172.21.15.45:42782) 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35ed215640 1 -- 172.21.15.45:0/3070581041 learned_addr learned my addr 172.21.15.45:0/3070581041 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 client.?.objecter ms_handle_connect 0x7f35e8135950 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 client.?.objecter ms_handle_connect 0x7f35e813c010 2024-09-16T03:05:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129065122 0 0) 0x7f35e8073460 con 0x7f35e813c010 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): my global_id is 4433 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35cc004ec0 con 0x7f35e813c010 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 406020988 0 0) 0x7f35e8070e10 con 0x7f35e8135950 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): my global_id is 4407 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35e8073460 con 0x7f35e8135950 2024-09-16T03:05:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 client.?.objecter ms_handle_connect 0x7f35e809fc40 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3325132790 0 0) 0x7f35cc004ec0 con 0x7f35e813c010 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35e8070e10 con 0x7f35e813c010 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f35e0001f90 con 0x7f35e813c010 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): dump: 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3473654226 0 0) 0x7f35e8073180 con 0x7f35e809fc40 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): my global_id is 4420 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35cc004ec0 con 0x7f35e809fc40 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3155258090 0 0) 0x7f35e8073460 con 0x7f35e8135950 2024-09-16T03:05:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35e8073180 con 0x7f35e8135950 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f35d80013c0 con 0x7f35e8135950 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): dump: 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2433670442 0 0) 0x7f35e8070e10 con 0x7f35e813c010 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 >> v1:172.21.15.46:6789/0 conn(0x7f35e809fc40 legacy=0x7f35e806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 >> v1:172.21.15.45:6789/0 conn(0x7f35e8135950 legacy=0x7f35e8070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 monclient: found mon.c 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e8073740 con 0x7f35e813c010 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 5 monclient: authenticate success, global_id 4433 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient: _renew_subs 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35e8073a20 con 0x7f35e813c010 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient: _renew_subs 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f35e8074180 con 0x7f35e813c010 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 librados: init done 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35e0001ba0 con 0x7f35e813c010 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f35e0004f40 con 0x7f35e813c010 2024-09-16T03:05:05.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35dffff640 4 set_mon_vals no callback set 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: dump: 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.901 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35d00065d0 con 0x7f35d0002870 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 client.4433.objecter ms_handle_connect 0x7f35d0002870 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 client.4433.objecter resend_mon_ops 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f35e0010fb0 con 0x7f35e813c010 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92143982 0 0) 0x7f35d00065d0 con 0x7f35d0002870 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient(hunting): my global_id is 4410 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35e8070e10 con 0x7f35d0002870 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f35e004b1a0 con 0x7f35e813c010 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 client.4433.objecter ms_dispatch 0x7f35e8000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 3 client.4433.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 3 client.4433.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 20 client.4433.objecter dump_active .. 0 homeless 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1282699902 0 0) 0x7f35e8070e10 con 0x7f35d0002870 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35d00065d0 con 0x7f35d0002870 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f35e4001650 con 0x7f35d0002870 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35de7fc640 10 monclient: dump: 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.903 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.878+0000 7f35ed215640 10 monclient: get_auth_request con 0x7f35cc03e260 auth_method 0 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.882+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1216454224 0 0) 0x7f35d00065d0 con 0x7f35d0002870 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.882+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35d000a170 con 0x7f35d0002870 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.882+0000 7f35de7fc640 10 client.4433.objecter ms_handle_connect 0x7f35cc03e260 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.882+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f35e4001970 con 0x7f35d0002870 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.882+0000 7f35de7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.882+0000 7f35de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:05.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.882+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 >> v1:172.21.15.45:6789/0 conn(0x7f35d0002870 legacy=0x7f35d0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.882+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 >> v1:172.21.15.45:6789/0 conn(0x7f35d0002870 legacy=0x7f35d0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:05.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:05:05.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35d0009e80 con 0x7f35d0009140 2024-09-16T03:05:05.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 client.4433.objecter ms_handle_connect 0x7f35d0009140 2024-09-16T03:05:05.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 client.4433.objecter resend_mon_ops 2024-09-16T03:05:05.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3807568457 0 0) 0x7f35d0009e80 con 0x7f35d0009140 2024-09-16T03:05:05.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:05.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 monclient(hunting): my global_id is 4413 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35d00065d0 con 0x7f35d0009140 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2654738837 0 0) 0x7f35d00065d0 con 0x7f35d0009140 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35d0009e80 con 0x7f35d0009140 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f35e40037a0 con 0x7f35d0009140 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 monclient: dump: 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:05.907 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1734743420 0 0) 0x7f35d0009e80 con 0x7f35d0009140 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f35d000c420 con 0x7f35d0009140 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4136 (unknown 0 0 2604282541) 0x7f35e4005980 con 0x7f35d0009140 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:05.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 >> v1:172.21.15.45:6789/0 conn(0x7f35d0009140 legacy=0x7f35d00095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35de7fc640 1 -- 172.21.15.45:0/3070581041 >> v1:172.21.15.45:6789/0 conn(0x7f35d0009140 legacy=0x7f35d00095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 4, 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 27735, 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:05:05.909 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 27, 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:05.910 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:05.911 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:05:05.912 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:05.913 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:05.914 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:05.915 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:05:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:05.920 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:05:05.921 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:05.921 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:05.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:05.921 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:05:05.921 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:05:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 10 librados: watch_flush enter 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 10 librados: watch_flush exit 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 20 client.4433.objecter shutdown clearing up homeless session... 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 10 client.4433.objecter successfully canceled tick 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 >> v1:172.21.15.46:6812/149360280 conn(0x7f35cc03e260 legacy=0x7f35cc040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 10 monclient: shutdown 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 >> v1:172.21.15.45:6790/0 conn(0x7f35e813c010 legacy=0x7f35e81d4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35ee217640 1 -- 172.21.15.45:0/3070581041 reap_dead start 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35ee217640 1 -- 172.21.15.45:0/3070581041 reap_dead start 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 shutdown_connections 2024-09-16T03:05:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 >> 172.21.15.45:0/3070581041 conn(0x7f35e80a2bd0 msgr2=0x7f35e813aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:05.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 shutdown_connections 2024-09-16T03:05:05.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 1 -- 172.21.15.45:0/3070581041 wait complete. 2024-09-16T03:05:05.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:05.886+0000 7f35efc7a640 1 librados: shutdown 2024-09-16T03:05:05.924 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:05:05.924 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:05:05.924 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient: get_monmap_and_config 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient: build_initial_monmap 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient: monmap: 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:06.040521+0000 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:06.040521+0000 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.041 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:06.042 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:06.042 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:06.042 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 Processor -- start 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 -- start start 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient: init 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient(hunting): picked mon.noname-a con 0x7f9b28137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient(hunting): picked mon.noname-b con 0x7f9b28138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient(hunting): picked mon.noname-c con 0x7f9b2813e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b280a4f90 con 0x7f9b28137d20 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b280a4cb0 con 0x7f9b2813e2a0 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b2809dc70 con 0x7f9b28138100 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient(hunting): authenticate will time out at 996.689816s 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2d500640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9b28137d20 0x7f9b280a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42798/0 (socket says 172.21.15.45:42798) 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2d500640 1 -- 172.21.15.45:0/3365147895 learned_addr learned my addr 172.21.15.45:0/3365147895 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2ccff640 1 --1- 172.21.15.45:0/3365147895 >> v1:172.21.15.45:6790/0 conn(0x7f9b28138100 0x7f9b2813cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35474/0 (socket says 172.21.15.45:35474) 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516598276 0 0) 0x7f9b2809dc70 con 0x7f9b28138100 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient(hunting): my global_id is 4436 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b14003530 con 0x7f9b28138100 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4157809740 0 0) 0x7f9b280a4f90 con 0x7f9b28137d20 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient(hunting): my global_id is 4416 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b2809dc70 con 0x7f9b28137d20 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3761568942 0 0) 0x7f9b2809dc70 con 0x7f9b28137d20 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 >> v1:172.21.15.45:6790/0 conn(0x7f9b28138100 legacy=0x7f9b2813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 >> v1:172.21.15.46:6789/0 conn(0x7f9b2813e2a0 legacy=0x7f9b28140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 monclient: found mon.noname-a 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b2809cc50 con 0x7f9b28137d20 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: _finish_auth 0 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:05:06.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b180015b0 con 0x7f9b28137d20 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: dump: 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.048 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 5 monclient: authenticate success, global_id 4416 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9b18001750 con 0x7f9b28137d20 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 1 -- 172.21.15.45:0/3365147895 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b18001be0 con 0x7f9b28137d20 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b1ffff640 10 monclient: dump: 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 4 set_mon_vals no callback set 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 10 monclient: shutdown 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 -- 172.21.15.45:0/3365147895 >> v1:172.21.15.45:6789/0 conn(0x7f9b28137d20 legacy=0x7f9b280a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 -- 172.21.15.45:0/3365147895 shutdown_connections 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 -- 172.21.15.45:0/3365147895 >> 172.21.15.45:0/3365147895 conn(0x7f9b280aa110 msgr2=0x7f9b280aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 -- 172.21.15.45:0/3365147895 shutdown_connections 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.038+0000 7f9b2f764640 1 -- 172.21.15.45:0/3365147895 wait complete. 2024-09-16T03:05:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient: build_initial_monmap 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient: monmap: 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 librados: starting msgr at 2024-09-16T03:05:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 librados: starting objecter 2024-09-16T03:05:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 Processor -- start 2024-09-16T03:05:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 -- start start 2024-09-16T03:05:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 librados: setting wanted keys 2024-09-16T03:05:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 librados: calling monclient init 2024-09-16T03:05:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient: init 2024-09-16T03:05:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient(hunting): picked mon.noname-b con 0x7f9b28137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient(hunting): picked mon.noname-a con 0x7f9b28138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient(hunting): picked mon.noname-c con 0x7f9b2813e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b2813bbf0 con 0x7f9b28138100 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b28139770 con 0x7f9b28137d20 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b280a1300 con 0x7f9b2813e2a0 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient(hunting): authenticate will time out at 996.693012s 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2ccff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9b28138100 0x7f9b28139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42808/0 (socket says 172.21.15.45:42808) 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2ccff640 1 -- 172.21.15.45:0/2365094642 learned_addr learned my addr 172.21.15.45:0/2365094642 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.?.objecter ms_handle_connect 0x7f9b2813e2a0 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.?.objecter ms_handle_connect 0x7f9b28138100 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1483913471 0 0) 0x7f9b280a1300 con 0x7f9b2813e2a0 2024-09-16T03:05:06.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): my global_id is 4439 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b0c004ec0 con 0x7f9b2813e2a0 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577949964 0 0) 0x7f9b2813bbf0 con 0x7f9b28138100 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): my global_id is 4419 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b280a1300 con 0x7f9b28138100 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.?.objecter ms_handle_connect 0x7f9b28137d20 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1255424962 0 0) 0x7f9b0c004ec0 con 0x7f9b2813e2a0 2024-09-16T03:05:06.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b2813bbf0 con 0x7f9b2813e2a0 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b20001c00 con 0x7f9b2813e2a0 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): dump: 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.060 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3704076703 0 0) 0x7f9b280a1300 con 0x7f9b28138100 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b0c004ec0 con 0x7f9b28138100 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b240013e0 con 0x7f9b28138100 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): dump: 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.061 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423522018 0 0) 0x7f9b28139770 con 0x7f9b28137d20 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): my global_id is 4426 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b280a1300 con 0x7f9b28137d20 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3230080462 0 0) 0x7f9b2813bbf0 con 0x7f9b2813e2a0 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 >> v1:172.21.15.46:6789/0 conn(0x7f9b28137d20 legacy=0x7f9b28138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 >> v1:172.21.15.45:6789/0 conn(0x7f9b28138100 legacy=0x7f9b28139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 monclient: found mon.c 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b2813b910 con 0x7f9b2813e2a0 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: _finish_auth 0 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 5 monclient: authenticate success, global_id 4439 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient: _renew_subs 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9b2813a830 con 0x7f9b2813e2a0 2024-09-16T03:05:06.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient: _renew_subs 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9b2813a3f0 con 0x7f9b2813e2a0 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 librados: init done 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9b20003980 con 0x7f9b2813e2a0 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b20003cd0 con 0x7f9b2813e2a0 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: dump: 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.064 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1f7fe640 4 set_mon_vals no callback set 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9b080065d0 con 0x7f9b08002870 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f9b2000fd40 con 0x7f9b2813e2a0 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.4439.objecter ms_handle_connect 0x7f9b08002870 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.4439.objecter resend_mon_ops 2024-09-16T03:05:06.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f9b2004a110 con 0x7f9b2813e2a0 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.4439.objecter ms_dispatch 0x7f9b28000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 3 client.4439.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 3 client.4439.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 20 client.4439.objecter dump_active .. 0 homeless 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354487098 0 0) 0x7f9b080065d0 con 0x7f9b08002870 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient(hunting): my global_id is 4442 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b2813bbf0 con 0x7f9b08002870 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b2ccff640 10 monclient: get_auth_request con 0x7f9b0c03e290 auth_method 0 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4275574634 0 0) 0x7f9b2813bbf0 con 0x7f9b08002870 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b080065d0 con 0x7f9b08002870 2024-09-16T03:05:06.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b18001820 con 0x7f9b08002870 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: dump: 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 953568073 0 0) 0x7f9b080065d0 con 0x7f9b08002870 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b0800a170 con 0x7f9b08002870 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 client.4439.objecter ms_handle_connect 0x7f9b0c03e290 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9b18001a40 con 0x7f9b08002870 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 >> v1:172.21.15.45:6790/0 conn(0x7f9b08002870 legacy=0x7f9b08004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.042+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 >> v1:172.21.15.45:6790/0 conn(0x7f9b08002870 legacy=0x7f9b08004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9b08009e60 con 0x7f9b08009120 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 client.4439.objecter ms_handle_connect 0x7f9b08009120 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 client.4439.objecter resend_mon_ops 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 555124661 0 0) 0x7f9b08009e60 con 0x7f9b08009120 2024-09-16T03:05:06.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 monclient(hunting): my global_id is 4445 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b080065d0 con 0x7f9b08009120 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3611933445 0 0) 0x7f9b080065d0 con 0x7f9b08009120 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b08009e60 con 0x7f9b08009120 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b18004990 con 0x7f9b08009120 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 monclient: dump: 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.069 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 490491028 0 0) 0x7f9b08009e60 con 0x7f9b08009120 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9b0800c400 con 0x7f9b08009120 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4134 (unknown 0 0 371220022) 0x7f9b18004920 con 0x7f9b08009120 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 >> v1:172.21.15.45:6790/0 conn(0x7f9b08009120 legacy=0x7f9b080095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b1dffb640 1 -- 172.21.15.45:0/2365094642 >> v1:172.21.15.45:6790/0 conn(0x7f9b08009120 legacy=0x7f9b080095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:06.071 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:05:06.071 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:05:06.071 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:05:06.071 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:05:06.071 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 4, 2024-09-16T03:05:06.071 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 27903, 2024-09-16T03:05:06.071 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:05:06.071 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:05:06.071 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 27, 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:06.072 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:06.073 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:05:06.074 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:06.075 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.076 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.077 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:06.078 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:05:06.079 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:06.080 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:05:06.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.082 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:05:06.083 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:05:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 10 librados: watch_flush enter 2024-09-16T03:05:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 10 librados: watch_flush exit 2024-09-16T03:05:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 20 client.4439.objecter shutdown clearing up homeless session... 2024-09-16T03:05:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 10 client.4439.objecter successfully canceled tick 2024-09-16T03:05:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 >> v1:172.21.15.46:6812/149360280 conn(0x7f9b0c03e290 legacy=0x7f9b0c040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 10 monclient: shutdown 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 >> v1:172.21.15.45:6790/0 conn(0x7f9b2813e2a0 legacy=0x7f9b281d4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2dd01640 1 -- 172.21.15.45:0/2365094642 reap_dead start 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2dd01640 1 -- 172.21.15.45:0/2365094642 reap_dead start 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 shutdown_connections 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 >> 172.21.15.45:0/2365094642 conn(0x7f9b280aa110 msgr2=0x7f9b280746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 shutdown_connections 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 1 -- 172.21.15.45:0/2365094642 wait complete. 2024-09-16T03:05:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.050+0000 7f9b2f764640 1 librados: shutdown 2024-09-16T03:05:06.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:05:06.086 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:05:06.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: get_monmap_and_config 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: build_initial_monmap 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: monmap: 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:06.204356+0000 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:06.204356+0000 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:06.206 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 Processor -- start 2024-09-16T03:05:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- start start 2024-09-16T03:05:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: init 2024-09-16T03:05:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient(hunting): picked mon.noname-c con 0x7fa8280a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient(hunting): picked mon.noname-b con 0x7fa8280a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient(hunting): picked mon.noname-a con 0x7fa828073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82809d9e0 con 0x7fa828073d40 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82809dc20 con 0x7fa8280a4d20 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa828073360 con 0x7fa8280a5810 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient(hunting): authenticate will time out at 996.853722s 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa825d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fa8280a5810 0x7fa82813cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35510/0 (socket says 172.21.15.45:35510) 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa825d9b640 1 -- 172.21.15.45:0/1626036338 learned_addr learned my addr 172.21.15.45:0/1626036338 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3478265920 0 0) 0x7fa828073360 con 0x7fa8280a5810 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient(hunting): my global_id is 4448 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa808003530 con 0x7fa8280a5810 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341986379 0 0) 0x7fa82809d9e0 con 0x7fa828073d40 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient(hunting): my global_id is 4422 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa828073360 con 0x7fa828073d40 2024-09-16T03:05:06.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1158572759 0 0) 0x7fa808003530 con 0x7fa8280a5810 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 >> v1:172.21.15.46:6789/0 conn(0x7fa8280a4d20 legacy=0x7fa8280a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 >> v1:172.21.15.45:6789/0 conn(0x7fa828073d40 legacy=0x7fa828140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: _finish_hunting 0 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 monclient: found mon.noname-b 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82809cf30 con 0x7fa8280a5810 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: _finish_auth 0 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: _check_auth_tickets 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa8100016a0 con 0x7fa8280a5810 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: dump: 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 5 monclient: authenticate success, global_id 4448 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:06.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa810001840 con 0x7fa8280a5810 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 4 set_mon_vals no callback set 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 1 -- 172.21.15.45:0/1626036338 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa810001cd0 con 0x7fa8280a5810 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82559a640 10 monclient: dump: 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: shutdown 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- 172.21.15.45:0/1626036338 >> v1:172.21.15.45:6790/0 conn(0x7fa8280a5810 legacy=0x7fa82813cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- 172.21.15.45:0/1626036338 shutdown_connections 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- 172.21.15.45:0/1626036338 >> 172.21.15.45:0/1626036338 conn(0x7fa8280aa720 msgr2=0x7fa8280aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- 172.21.15.45:0/1626036338 shutdown_connections 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- 172.21.15.45:0/1626036338 wait complete. 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: build_initial_monmap 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: monmap: 2024-09-16T03:05:06.219 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 librados: starting msgr at 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 librados: starting objecter 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 Processor -- start 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 -- start start 2024-09-16T03:05:06.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 librados: setting wanted keys 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 1 librados: calling monclient init 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.202+0000 7fa82cd99640 10 monclient: init 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient(hunting): picked mon.noname-b con 0x7fa828073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient(hunting): picked mon.noname-a con 0x7fa8280a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient(hunting): picked mon.noname-c con 0x7fa8280a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa828140310 con 0x7fa8280a4d20 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82813d6e0 con 0x7fa828073d40 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8281405f0 con 0x7fa8280a5810 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient(hunting): authenticate will time out at 996.856463s 2024-09-16T03:05:06.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa825d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa8280a4d20 0x7fa82813cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42838/0 (socket says 172.21.15.45:42838) 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa825d9b640 1 -- 172.21.15.45:0/1066971762 learned_addr learned my addr 172.21.15.45:0/1066971762 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.?.objecter ms_handle_connect 0x7fa8280a5810 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.?.objecter ms_handle_connect 0x7fa8280a4d20 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.?.objecter ms_handle_connect 0x7fa828073d40 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1817320485 0 0) 0x7fa8281405f0 con 0x7fa8280a5810 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): my global_id is 4451 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa80c004ef0 con 0x7fa8280a5810 2024-09-16T03:05:06.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932457160 0 0) 0x7fa828140310 con 0x7fa8280a4d20 2024-09-16T03:05:06.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): my global_id is 4425 2024-09-16T03:05:06.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8281405f0 con 0x7fa8280a4d20 2024-09-16T03:05:06.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183517445 0 0) 0x7fa82813d6e0 con 0x7fa828073d40 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): my global_id is 4432 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa828140310 con 0x7fa828073d40 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3948795742 0 0) 0x7fa80c004ef0 con 0x7fa8280a5810 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa82813d6e0 con 0x7fa8280a5810 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa818001940 con 0x7fa8280a5810 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): dump: 2024-09-16T03:05:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2737639921 0 0) 0x7fa8281405f0 con 0x7fa8280a4d20 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa80c004ef0 con 0x7fa8280a4d20 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa810001b90 con 0x7fa8280a4d20 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): dump: 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3748818479 0 0) 0x7fa82813d6e0 con 0x7fa8280a5810 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 >> v1:172.21.15.46:6789/0 conn(0x7fa828073d40 legacy=0x7fa82813c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 >> v1:172.21.15.45:6789/0 conn(0x7fa8280a4d20 legacy=0x7fa82813cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 monclient: found mon.c 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8280a1cf0 con 0x7fa8280a5810 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: _finish_auth 0 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 5 monclient: authenticate success, global_id 4451 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient: _renew_subs 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8280a1510 con 0x7fa8280a5810 2024-09-16T03:05:06.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient: _renew_subs 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa82813f0c0 con 0x7fa8280a5810 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 1 librados: init done 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa818003620 con 0x7fa8280a5810 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa818003970 con 0x7fa8280a5810 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa824d99640 4 set_mon_vals no callback set 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: dump: 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.228 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa8000065d0 con 0x7fa800002870 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fa81800f9e0 con 0x7fa8280a5810 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7fa818049e20 con 0x7fa8280a5810 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.4451.objecter ms_dispatch 0x7fa828000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:06.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 3 client.4451.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 3 client.4451.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 20 client.4451.objecter dump_active .. 0 homeless 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.4451.objecter ms_handle_connect 0x7fa800002870 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.4451.objecter resend_mon_ops 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968625960 0 0) 0x7fa8000065d0 con 0x7fa800002870 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient(hunting): my global_id is 4435 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa82813d6e0 con 0x7fa800002870 2024-09-16T03:05:06.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa825d9b640 10 monclient: get_auth_request con 0x7fa80c03e1e0 auth_method 0 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3567407422 0 0) 0x7fa82813d6e0 con 0x7fa800002870 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8000065d0 con 0x7fa800002870 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa820001ba0 con 0x7fa800002870 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 monclient: dump: 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.232 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.233 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.233 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.233 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.233 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 10 client.4451.objecter ms_handle_connect 0x7fa80c03e1e0 2024-09-16T03:05:06.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 298352276 0 0) 0x7fa8000065d0 con 0x7fa800002870 2024-09-16T03:05:06.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.206+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa80000a170 con 0x7fa800002870 2024-09-16T03:05:06.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.210+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa820001dc0 con 0x7fa800002870 2024-09-16T03:05:06.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.210+0000 7fa81effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:06.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.210+0000 7fa81effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:06.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.210+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 >> v1:172.21.15.46:6789/0 conn(0x7fa800002870 legacy=0x7fa800004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.210+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 >> v1:172.21.15.46:6789/0 conn(0x7fa800002870 legacy=0x7fa800004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:06.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa82cd99640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:05:06.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa800009e60 con 0x7fa800009120 2024-09-16T03:05:06.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 10 client.4451.objecter ms_handle_connect 0x7fa800009120 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 10 client.4451.objecter resend_mon_ops 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229046281 0 0) 0x7fa800009e60 con 0x7fa800009120 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 10 monclient(hunting): my global_id is 4438 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8000065d0 con 0x7fa800009120 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3046018164 0 0) 0x7fa8000065d0 con 0x7fa800009120 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa800009e60 con 0x7fa800009120 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa820003b20 con 0x7fa800009120 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:06.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 10 monclient: dump: 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 445909464 0 0) 0x7fa800009e60 con 0x7fa800009120 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.214+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa80000c400 con 0x7fa800009120 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4301 (unknown 0 0 1542957171) 0x7fa820003b30 con 0x7fa800009120 2024-09-16T03:05:06.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa81effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa81effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 >> v1:172.21.15.46:6789/0 conn(0x7fa800009120 legacy=0x7fa8000095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa81effd640 1 -- 172.21.15.45:0/1066971762 >> v1:172.21.15.46:6789/0 conn(0x7fa800009120 legacy=0x7fa8000095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 4, 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 28023, 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:05:06.238 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 27, 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:06.239 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:06.240 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:05:06.241 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:06.242 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.243 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.244 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:06.245 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:05:06.246 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:06.248 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:06.249 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:06.250 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:05:06.250 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:06.250 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:06.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:06.250 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:05:06.250 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 10 librados: watch_flush enter 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 10 librados: watch_flush exit 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 20 client.4451.objecter shutdown clearing up homeless session... 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 10 client.4451.objecter successfully canceled tick 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 >> v1:172.21.15.46:6812/149360280 conn(0x7fa80c03e1e0 legacy=0x7fa80c0406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 10 monclient: shutdown 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 >> v1:172.21.15.45:6790/0 conn(0x7fa8280a5810 legacy=0x7fa8281d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa826d9d640 1 -- 172.21.15.45:0/1066971762 reap_dead start 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa826d9d640 1 -- 172.21.15.45:0/1066971762 reap_dead start 2024-09-16T03:05:06.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 shutdown_connections 2024-09-16T03:05:06.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 >> 172.21.15.45:0/1066971762 conn(0x7fa8280aa720 msgr2=0x7fa8280a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:06.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 shutdown_connections 2024-09-16T03:05:06.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 1 -- 172.21.15.45:0/1066971762 wait complete. 2024-09-16T03:05:06.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:06.218+0000 7fa82cd99640 1 librados: shutdown 2024-09-16T03:05:06.253 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:05:06.253 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T03:05:06.253 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T03:05:06.253 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T03:05:06.253 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:05:06.254 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:05:06.254 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:05:06.264 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:05:06.264 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:05:06.264 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:05:06.264 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:05:21.266 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:05:21.267 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:05:21.267 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:05:21.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:05:21.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient: get_monmap_and_config 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient: build_initial_monmap 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient: monmap: 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:21.398204+0000 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:21.398204+0000 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:21.399 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:21.400 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:21.400 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 1 Processor -- start 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 1 -- start start 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient: init 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient(hunting): picked mon.noname-b con 0x7fbba0135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient(hunting): picked mon.noname-c con 0x7fbba009fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient(hunting): picked mon.noname-a con 0x7fbba013c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba009f7e0 con 0x7fbba013c010 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba009b160 con 0x7fbba009fc40 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba009b440 con 0x7fbba0135950 2024-09-16T03:05:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba8175640 10 monclient(hunting): authenticate will time out at 1012.047565s 2024-09-16T03:05:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba6712640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbba013e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba5f11640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fbba0135950 0x7fbba0135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35970/0 (socket says 172.21.15.45:35970) 2024-09-16T03:05:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.394+0000 7fbba5f11640 1 -- 172.21.15.45:0/1881050186 learned_addr learned my addr 172.21.15.45:0/1881050186 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:21.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:21.598+0000 7fbba6712640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbba013e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:22.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:22.002+0000 7fbba6712640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbba013e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:22.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:22.806+0000 7fbba6712640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbba013e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:24.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 10 monclient(hunting): tick 2024-09-16T03:05:24.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:05:24.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:05:24.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:05:24.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6790/0 conn(0x7fbba0135950 legacy=0x7fbba0135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:24.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.46:6789/0 conn(0x7fbba009fc40 legacy=0x7fbba013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:24.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbba013e400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:05:24.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:24.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fbba013c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:24.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fbb8c006e90 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:24.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fbba009fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:24.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 1 -- 172.21.15.45:0/1881050186 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba0136440 con 0x7fbba013c010 2024-09-16T03:05:24.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 1 -- 172.21.15.45:0/1881050186 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba009f7e0 con 0x7fbba009fc40 2024-09-16T03:05:24.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbba5f11640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbb8c005750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:24.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.394+0000 7fbb9f7fe640 1 -- 172.21.15.45:0/1881050186 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb8c010430 con 0x7fbb8c006e90 2024-09-16T03:05:24.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.398+0000 7fbb9f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:24.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:24.602+0000 7fbba5f11640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbb8c005750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:25.006+0000 7fbba5f11640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbb8c005750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:25.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:25.810+0000 7fbba5f11640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbb8c005750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 -- 172.21.15.45:0/1881050186 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1752426682 0 0) 0x7fbb8c010430 con 0x7fbb8c006e90 2024-09-16T03:05:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient(hunting): my global_id is 4454 2024-09-16T03:05:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 -- 172.21.15.45:0/1881050186 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb88003530 con 0x7fbb8c006e90 2024-09-16T03:05:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 -- 172.21.15.45:0/1881050186 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2400406382 0 0) 0x7fbb88003530 con 0x7fbb8c006e90 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.46:6789/0 conn(0x7fbba009fc40 legacy=0x7fbb8c00cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6789/0 conn(0x7fbba013c010 legacy=0x7fbb8c005750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: _finish_hunting 0 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 monclient: found mon.noname-b 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 -- 172.21.15.45:0/1881050186 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8c0116d0 con 0x7fbb8c006e90 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: _finish_auth 0 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: _check_auth_tickets 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 -- 172.21.15.45:0/1881050186 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb94001960 con 0x7fbb8c006e90 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: dump: 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 5 monclient: authenticate success, global_id 4454 2024-09-16T03:05:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 -- 172.21.15.45:0/1881050186 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb94001b00 con 0x7fbb8c006e90 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 1 -- 172.21.15.45:0/1881050186 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb94001ed0 con 0x7fbb8c006e90 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 4 set_mon_vals no callback set 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba4f0f640 10 monclient: dump: 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 10 monclient: shutdown 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 1 -- 172.21.15.45:0/1881050186 >> v1:172.21.15.45:6790/0 conn(0x7fbb8c006e90 legacy=0x7fbb8c009320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 1 -- 172.21.15.45:0/1881050186 shutdown_connections 2024-09-16T03:05:26.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 1 -- 172.21.15.45:0/1881050186 >> 172.21.15.45:0/1881050186 conn(0x7fbba00a2bd0 msgr2=0x7fbba00a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:26.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 1 -- 172.21.15.45:0/1881050186 shutdown_connections 2024-09-16T03:05:26.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.310+0000 7fbba8175640 1 -- 172.21.15.45:0/1881050186 wait complete. 2024-09-16T03:05:26.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient: build_initial_monmap 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient: monmap: 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 1 librados: starting msgr at 2024-09-16T03:05:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 1 librados: starting objecter 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 1 Processor -- start 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 1 -- start start 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 1 librados: setting wanted keys 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 1 librados: calling monclient init 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient: init 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient(hunting): picked mon.noname-a con 0x7fbb8c006e90 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient(hunting): picked mon.noname-b con 0x7fbba009fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient(hunting): picked mon.noname-c con 0x7fbba013c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba5f11640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fbb8c006e90 legacy=0x7fbba0072690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba0134cd0 con 0x7fbb8c006e90 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba012cdd0 con 0x7fbba009fc40 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba009d2c0 con 0x7fbba013c010 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba8175640 10 monclient(hunting): authenticate will time out at 1016.966967s 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba6712640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fbba013c010 0x7fbba01d4ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36000/0 (socket says 172.21.15.45:36000) 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbba6712640 1 -- 172.21.15.45:0/74491904 learned_addr learned my addr 172.21.15.45:0/74491904 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 10 client.?.objecter ms_handle_connect 0x7fbba013c010 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 10 client.?.objecter ms_handle_connect 0x7fbba009fc40 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228459529 0 0) 0x7fbba009d2c0 con 0x7fbba013c010 2024-09-16T03:05:26.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 10 monclient(hunting): my global_id is 4457 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.314+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb8c00a2b0 con 0x7fbba013c010 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4084118561 0 0) 0x7fbb8c00a2b0 con 0x7fbba013c010 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbba009d2c0 con 0x7fbba013c010 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb90004270 con 0x7fbba013c010 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 10 monclient(hunting): dump: 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159411128 0 0) 0x7fbba012cdd0 con 0x7fbba009fc40 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 10 monclient(hunting): my global_id is 4441 2024-09-16T03:05:26.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.318+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb8c00a2b0 con 0x7fbba009fc40 2024-09-16T03:05:26.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1513994013 0 0) 0x7fbb8c00a2b0 con 0x7fbba009fc40 2024-09-16T03:05:26.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbba012cdd0 con 0x7fbba009fc40 2024-09-16T03:05:26.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb94001e50 con 0x7fbba009fc40 2024-09-16T03:05:26.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:26.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:26.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient(hunting): dump: 2024-09-16T03:05:26.331 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.331 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.331 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.332 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.332 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.332 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.332 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.332 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.332 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.332 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 384527725 0 0) 0x7fbba009d2c0 con 0x7fbba013c010 2024-09-16T03:05:26.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 >> v1:172.21.15.46:6789/0 conn(0x7fbba009fc40 legacy=0x7fbba01d17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 >> v1:172.21.15.45:6789/0 conn(0x7fbb8c006e90 legacy=0x7fbba0072690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 monclient: found mon.c 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba012c810 con 0x7fbba013c010 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 5 monclient: authenticate success, global_id 4457 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 10 monclient: _renew_subs 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbba012caf0 con 0x7fbba013c010 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 10 monclient: _renew_subs 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbba0072f70 con 0x7fbba013c010 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 1 librados: init done 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb90004780 con 0x7fbba013c010 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb90004ad0 con 0x7fbba013c010 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9ffff640 4 set_mon_vals no callback set 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 monclient: dump: 2024-09-16T03:05:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fbb90010bc0 con 0x7fbba013c010 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb80005090 con 0x7fbba013c010 2024-09-16T03:05:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7fbb9004b160 con 0x7fbba013c010 2024-09-16T03:05:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 client.4457.objecter ms_dispatch 0x7fbba0000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 3 client.4457.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 3 client.4457.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 20 client.4457.objecter dump_active .. 0 homeless 2024-09-16T03:05:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbba5f11640 10 monclient: get_auth_request con 0x7fbb8c03e7c0 auth_method 0 2024-09-16T03:05:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.322+0000 7fbb9e7fc640 10 client.4457.objecter ms_handle_connect 0x7fbb8c03e7c0 2024-09-16T03:05:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.326+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fbb9001aea0 con 0x7fbba013c010 2024-09-16T03:05:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.326+0000 7fbb9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.326+0000 7fbb9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:26.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.570+0000 7fbba8175640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:05:26.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.570+0000 7fbba8175640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:05:26.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.570+0000 7fbba8175640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.570+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbb80005380 con 0x7fbba013c010 2024-09-16T03:05:26.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbb9e7fc640 1 -- 172.21.15.45:0/74491904 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1396 (unknown 254746657 0 3597089505) 0x7fbb9001fde0 con 0x7fbba013c010 2024-09-16T03:05:26.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbb9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:05:26.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbb9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:26.577 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":6,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:05:26.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 10 librados: watch_flush enter 2024-09-16T03:05:26.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 10 librados: watch_flush exit 2024-09-16T03:05:26.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 20 client.4457.objecter shutdown clearing up homeless session... 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 10 client.4457.objecter successfully canceled tick 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 >> v1:172.21.15.46:6812/149360280 conn(0x7fbb8c03e7c0 legacy=0x7fbb8c040c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 10 monclient: shutdown 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 >> v1:172.21.15.45:6790/0 conn(0x7fbba013c010 legacy=0x7fbba01d4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 shutdown_connections 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 >> 172.21.15.45:0/74491904 conn(0x7fbba00a2bd0 msgr2=0x7fbba00a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 shutdown_connections 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 1 -- 172.21.15.45:0/74491904 wait complete. 2024-09-16T03:05:26.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.574+0000 7fbba8175640 1 librados: shutdown 2024-09-16T03:05:26.597 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:05:26.597 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:05:26.597 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:05:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient: get_monmap_and_config 2024-09-16T03:05:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient: build_initial_monmap 2024-09-16T03:05:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient: monmap: 2024-09-16T03:05:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:26.713257+0000 2024-09-16T03:05:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:26.713257+0000 2024-09-16T03:05:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:26.715 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 1 Processor -- start 2024-09-16T03:05:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 1 -- start start 2024-09-16T03:05:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient: init 2024-09-16T03:05:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:05:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient(hunting): picked mon.noname-c con 0x7f53c8135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient(hunting): picked mon.noname-a con 0x7f53c809fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient(hunting): picked mon.noname-b con 0x7f53c813c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53c809f7e0 con 0x7f53c809fc40 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53c809b160 con 0x7f53c8135950 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53c809b440 con 0x7f53c813c010 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient(hunting): authenticate will time out at 1017.362564s 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c5d9b640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f53c809fc40 legacy=0x7f53c813a8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c6d9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f53c813c010 0x7f53c813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36010/0 (socket says 172.21.15.45:36010) 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c6d9d640 1 -- 172.21.15.45:0/2388354647 learned_addr learned my addr 172.21.15.45:0/2388354647 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3503387734 0 0) 0x7f53c809b440 con 0x7f53c813c010 2024-09-16T03:05:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient(hunting): my global_id is 4460 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53b0003530 con 0x7f53c813c010 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837218286 0 0) 0x7f53c809b160 con 0x7f53c8135950 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient(hunting): my global_id is 4444 2024-09-16T03:05:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53c809b440 con 0x7f53c8135950 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2865010511 0 0) 0x7f53b0003530 con 0x7f53c813c010 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 >> v1:172.21.15.46:6789/0 conn(0x7f53c8135950 legacy=0x7f53c8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 >> v1:172.21.15.45:6789/0 conn(0x7f53c809fc40 legacy=0x7f53c813a8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: _finish_hunting 0 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 monclient: found mon.noname-b 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c8136440 con 0x7f53c813c010 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: _finish_auth 0 2024-09-16T03:05:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: _check_auth_tickets 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f53c0001a20 con 0x7f53c813c010 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: dump: 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 5 monclient: authenticate success, global_id 4460 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53c0001bc0 con 0x7f53c813c010 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 1 -- 172.21.15.45:0/2388354647 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f53c0002010 con 0x7f53c813c010 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53cce8e640 4 set_mon_vals no callback set 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.710+0000 7f53c559a640 10 monclient: dump: 2024-09-16T03:05:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: shutdown 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- 172.21.15.45:0/2388354647 >> v1:172.21.15.45:6790/0 conn(0x7f53c813c010 legacy=0x7f53c813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- 172.21.15.45:0/2388354647 shutdown_connections 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- 172.21.15.45:0/2388354647 >> 172.21.15.45:0/2388354647 conn(0x7f53c80a2bd0 msgr2=0x7f53c80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- 172.21.15.45:0/2388354647 shutdown_connections 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- 172.21.15.45:0/2388354647 wait complete. 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: build_initial_monmap 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: monmap: 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:26.731 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 librados: starting msgr at 2024-09-16T03:05:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 librados: starting objecter 2024-09-16T03:05:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 Processor -- start 2024-09-16T03:05:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- start start 2024-09-16T03:05:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 librados: setting wanted keys 2024-09-16T03:05:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 librados: calling monclient init 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: init 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient(hunting): picked mon.noname-b con 0x7f53c809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient(hunting): picked mon.noname-c con 0x7f53c8135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient(hunting): picked mon.noname-a con 0x7f53c813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53c809ef90 con 0x7f53c813c010 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53c809f270 con 0x7f53c809fc40 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53c809da70 con 0x7f53c8135950 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient(hunting): authenticate will time out at 1017.366010s 2024-09-16T03:05:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53c5d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f53c8135950 0x7f53c81d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36014/0 (socket says 172.21.15.45:36014) 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53c6d9d640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f53c813c010 legacy=0x7f53c81d56d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53c5d9b640 1 -- 172.21.15.45:0/110013884 learned_addr learned my addr 172.21.15.45:0/110013884 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 client.?.objecter ms_handle_connect 0x7f53c8135950 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 client.?.objecter ms_handle_connect 0x7f53c809fc40 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 860754700 0 0) 0x7f53c809da70 con 0x7f53c8135950 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): my global_id is 4463 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53a8004ec0 con 0x7f53c8135950 2024-09-16T03:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794995703 0 0) 0x7f53c809f270 con 0x7f53c809fc40 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): my global_id is 4447 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53c809da70 con 0x7f53c809fc40 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 598056439 0 0) 0x7f53a8004ec0 con 0x7f53c8135950 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53c809f270 con 0x7f53c8135950 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f53bc004260 con 0x7f53c8135950 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): dump: 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.734 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1296904780 0 0) 0x7f53c809f270 con 0x7f53c8135950 2024-09-16T03:05:26.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 >> v1:172.21.15.46:6789/0 conn(0x7f53c809fc40 legacy=0x7f53c8099290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 >> v1:172.21.15.45:6789/0 conn(0x7f53c813c010 legacy=0x7f53c81d56d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 monclient: found mon.c 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c809d2c0 con 0x7f53c8135950 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 5 monclient: authenticate success, global_id 4463 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: _renew_subs 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53c813b6f0 con 0x7f53c8135950 2024-09-16T03:05:26.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:26.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: _renew_subs 2024-09-16T03:05:26.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f53c80737d0 con 0x7f53c8135950 2024-09-16T03:05:26.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 librados: init done 2024-09-16T03:05:26.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53bc004480 con 0x7f53c8135950 2024-09-16T03:05:26.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:26.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f53bc0048d0 con 0x7f53c8135950 2024-09-16T03:05:26.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53c4d99640 4 set_mon_vals no callback set 2024-09-16T03:05:26.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:26.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 monclient: dump: 2024-09-16T03:05:26.739 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.739 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.739 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f53ac0065d0 con 0x7f53ac002870 2024-09-16T03:05:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 client.4463.objecter ms_handle_connect 0x7f53ac002870 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 10 client.4463.objecter resend_mon_ops 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.714+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f53bc004a70 con 0x7f53c8135950 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f53bc04b150 con 0x7f53c8135950 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 client.4463.objecter ms_dispatch 0x7f53c8000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 3 client.4463.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 3 client.4463.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 20 client.4463.objecter dump_active .. 0 homeless 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325585429 0 0) 0x7f53ac0065d0 con 0x7f53ac002870 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 monclient(hunting): my global_id is 4466 2024-09-16T03:05:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53c809f270 con 0x7f53ac002870 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53c6d9d640 10 monclient: get_auth_request con 0x7f53a803e000 auth_method 0 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1647694386 0 0) 0x7f53c809f270 con 0x7f53ac002870 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53ac0065d0 con 0x7f53ac002870 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f53b80017f0 con 0x7f53ac002870 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 monclient: dump: 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1202841181 0 0) 0x7f53ac0065d0 con 0x7f53ac002870 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53ac00a170 con 0x7f53ac002870 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 client.4463.objecter ms_handle_connect 0x7f53a803e000 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f53b8001a50 con 0x7f53ac002870 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 >> v1:172.21.15.45:6790/0 conn(0x7f53ac002870 legacy=0x7f53ac004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.718+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 >> v1:172.21.15.45:6790/0 conn(0x7f53ac002870 legacy=0x7f53ac004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:26.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53cce8e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:05:26.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f53ac009e50 con 0x7f53ac009110 2024-09-16T03:05:26.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 client.4463.objecter ms_handle_connect 0x7f53ac009110 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 client.4463.objecter resend_mon_ops 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502867576 0 0) 0x7f53ac009e50 con 0x7f53ac009110 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 monclient(hunting): my global_id is 4469 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53ac0065d0 con 0x7f53ac009110 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3922493537 0 0) 0x7f53ac0065d0 con 0x7f53ac009110 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53ac009e50 con 0x7f53ac009110 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f53b80037c0 con 0x7f53ac009110 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:26.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 monclient: dump: 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 700235212 0 0) 0x7f53ac009e50 con 0x7f53ac009110 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53ac00c3b0 con 0x7f53ac009110 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4122 (unknown 0 0 821802990) 0x7f53b8004650 con 0x7f53ac009110 2024-09-16T03:05:26.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:05:26.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:26.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 >> v1:172.21.15.45:6790/0 conn(0x7f53ac009110 legacy=0x7f53ac0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.722+0000 7f53b6ffd640 1 -- 172.21.15.45:0/110013884 >> v1:172.21.15.45:6790/0 conn(0x7f53ac009110 legacy=0x7f53ac0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:26.748 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:05:26.748 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:05:26.748 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:05:26.748 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:05:26.748 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 6, 2024-09-16T03:05:26.748 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 48575, 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:26.749 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:26.750 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:26.751 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:26.751 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:26.751 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:26.751 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:26.751 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:26.751 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:26.753 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.755 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:26.756 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:05:26.757 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.758 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.759 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:26.760 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:26.761 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:05:26.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.761 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.761 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:05:26.761 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 10 librados: watch_flush enter 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 10 librados: watch_flush exit 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 20 client.4463.objecter shutdown clearing up homeless session... 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 10 client.4463.objecter successfully canceled tick 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 >> v1:172.21.15.46:6812/149360280 conn(0x7f53a803e000 legacy=0x7f53a80404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 10 monclient: shutdown 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 >> v1:172.21.15.45:6790/0 conn(0x7f53c8135950 legacy=0x7f53c81d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53c6d9d640 1 -- 172.21.15.45:0/110013884 reap_dead start 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53c6d9d640 1 -- 172.21.15.45:0/110013884 reap_dead start 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 shutdown_connections 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 >> 172.21.15.45:0/110013884 conn(0x7f53c80a2bd0 msgr2=0x7f53c813ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:26.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 shutdown_connections 2024-09-16T03:05:26.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 1 -- 172.21.15.45:0/110013884 wait complete. 2024-09-16T03:05:26.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.726+0000 7f53cce8e640 1 librados: shutdown 2024-09-16T03:05:26.764 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:05:26.764 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:05:26.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.874+0000 7f4b01975640 10 monclient: get_monmap_and_config 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.874+0000 7f4b01975640 10 monclient: build_initial_monmap 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.874+0000 7f4b01975640 10 monclient: monmap: 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:05:26.879546+0000 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:05:26.879546+0000 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:05:26.881 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:05:26.881 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 Processor -- start 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- start start 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient: init 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient(hunting): picked mon.noname-c con 0x7f4afc135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient(hunting): picked mon.noname-b con 0x7f4afc09fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient(hunting): picked mon.noname-a con 0x7f4afc13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc09f7e0 con 0x7f4afc13c010 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc09b160 con 0x7f4afc135950 2024-09-16T03:05:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc09b440 con 0x7f4afc09fc40 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient(hunting): authenticate will time out at 1017.528913s 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4afb7fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f4afc13c010 legacy=0x7f4afc13e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4afa7fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4afc09fc40 0x7f4afc13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36034/0 (socket says 172.21.15.45:36034) 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4afa7fc640 1 -- 172.21.15.45:0/3489670520 learned_addr learned my addr 172.21.15.45:0/3489670520 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1637501908 0 0) 0x7f4afc09b440 con 0x7f4afc09fc40 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient(hunting): my global_id is 4472 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4adc003530 con 0x7f4afc09fc40 2024-09-16T03:05:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2926421906 0 0) 0x7f4afc09b160 con 0x7f4afc135950 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient(hunting): my global_id is 4450 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4afc09b440 con 0x7f4afc135950 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1020446903 0 0) 0x7f4adc003530 con 0x7f4afc09fc40 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 >> v1:172.21.15.46:6789/0 conn(0x7f4afc135950 legacy=0x7f4afc135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 >> v1:172.21.15.45:6789/0 conn(0x7f4afc13c010 legacy=0x7f4afc13e400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 monclient: found mon.noname-b 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc136440 con 0x7f4afc09fc40 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 5 monclient: authenticate success, global_id 4472 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4ae40016a0 con 0x7f4afc09fc40 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: dump: 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ae4001840 con 0x7f4afc09fc40 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient: get_monmap_and_config success 2024-09-16T03:05:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 1 -- 172.21.15.45:0/3489670520 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4ae4001c90 con 0x7f4afc09fc40 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 4 set_mon_vals no callback set 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4af9ffb640 10 monclient: dump: 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient: shutdown 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- 172.21.15.45:0/3489670520 >> v1:172.21.15.45:6790/0 conn(0x7f4afc09fc40 legacy=0x7f4afc13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- 172.21.15.45:0/3489670520 shutdown_connections 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- 172.21.15.45:0/3489670520 >> 172.21.15.45:0/3489670520 conn(0x7f4afc0a2bd0 msgr2=0x7f4afc0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- 172.21.15.45:0/3489670520 shutdown_connections 2024-09-16T03:05:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- 172.21.15.45:0/3489670520 wait complete. 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient: build_initial_monmap 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient: monmap: 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:05:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:05:26.892 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 librados: starting msgr at 2024-09-16T03:05:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 librados: starting objecter 2024-09-16T03:05:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 Processor -- start 2024-09-16T03:05:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 -- start start 2024-09-16T03:05:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 librados: setting wanted keys 2024-09-16T03:05:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 1 librados: calling monclient init 2024-09-16T03:05:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.878+0000 7f4b01975640 10 monclient: init 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient: _reopen_session rank -1 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient(hunting): picked mon.noname-a con 0x7f4afc09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient(hunting): picked mon.noname-c con 0x7f4afc135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient(hunting): picked mon.noname-b con 0x7f4afc13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc13b6f0 con 0x7f4afc09fc40 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc071a00 con 0x7f4afc13c010 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc071ce0 con 0x7f4afc135950 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient(hunting): _renew_subs 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient(hunting): authenticate will time out at 1017.532109s 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4afaffd640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f4afc09fc40 legacy=0x7f4afc1cab30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:05:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4afa7fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4afc135950 0x7f4afc1cb240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36038/0 (socket says 172.21.15.45:36038) 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4afa7fc640 1 -- 172.21.15.45:0/660245148 learned_addr learned my addr 172.21.15.45:0/660245148 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 client.?.objecter ms_handle_connect 0x7f4afc135950 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963340327 0 0) 0x7f4afc071ce0 con 0x7f4afc135950 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): my global_id is 4475 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ae0004ec0 con 0x7f4afc135950 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 client.?.objecter ms_handle_connect 0x7f4afc13c010 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2861150485 0 0) 0x7f4afc071a00 con 0x7f4afc13c010 2024-09-16T03:05:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): my global_id is 4453 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4afc071ce0 con 0x7f4afc13c010 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4049666599 0 0) 0x7f4ae0004ec0 con 0x7f4afc135950 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4afc071a00 con 0x7f4afc135950 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4ae4001c50 con 0x7f4afc135950 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): dump: 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3491699328 0 0) 0x7f4afc071a00 con 0x7f4afc135950 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 >> v1:172.21.15.46:6789/0 conn(0x7f4afc13c010 legacy=0x7f4afc1d2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 >> v1:172.21.15.45:6789/0 conn(0x7f4afc09fc40 legacy=0x7f4afc1cab30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 monclient: found mon.c 2024-09-16T03:05:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc1cb950 con 0x7f4afc135950 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: _finish_auth 0 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 5 monclient: authenticate success, global_id 4475 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient: _renew_subs 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4afc13bbb0 con 0x7f4afc135950 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient: _renew_subs 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4afc06f590 con 0x7f4afc135950 2024-09-16T03:05:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 1 librados: init done 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ae4001200 con 0x7f4afc135950 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4ae4004aa0 con 0x7f4afc135950 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af97fa640 4 set_mon_vals no callback set 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: dump: 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4ad40065d0 con 0x7f4ad4002870 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4ae4004cc0 con 0x7f4afc135950 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 client.4475.objecter ms_handle_connect 0x7f4ad4002870 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 client.4475.objecter resend_mon_ops 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f4ae404afa0 con 0x7f4afc135950 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 client.4475.objecter ms_dispatch 0x7f4afc000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 3 client.4475.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:05:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 3 client.4475.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 20 client.4475.objecter dump_active .. 0 homeless 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884917766 0 0) 0x7f4ad40065d0 con 0x7f4ad4002870 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient(hunting): my global_id is 4456 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4afc071a00 con 0x7f4ad4002870 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4afb7fe640 10 monclient: get_auth_request con 0x7f4ae003e000 auth_method 0 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2911573338 0 0) 0x7f4afc071a00 con 0x7f4ad4002870 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ad40065d0 con 0x7f4ad4002870 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4af4001b50 con 0x7f4ad4002870 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: dump: 2024-09-16T03:05:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 client.4475.objecter ms_handle_connect 0x7f4ae003e000 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2771520837 0 0) 0x7f4ad40065d0 con 0x7f4ad4002870 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ad400a170 con 0x7f4ad4002870 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4af4005b50 con 0x7f4ad4002870 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 >> v1:172.21.15.46:6789/0 conn(0x7f4ad4002870 legacy=0x7f4ad4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.882+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 >> v1:172.21.15.46:6789/0 conn(0x7f4ad4002870 legacy=0x7f4ad4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4b01975640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4ad4009e80 con 0x7f4ad4009140 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 10 client.4475.objecter ms_handle_connect 0x7f4ad4009140 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 10 client.4475.objecter resend_mon_ops 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1936489126 0 0) 0x7f4ad4009e80 con 0x7f4ad4009140 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 10 monclient(hunting): my global_id is 4459 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ad40065d0 con 0x7f4ad4009140 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3257004253 0 0) 0x7f4ad40065d0 con 0x7f4ad4009140 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ad4009e80 con 0x7f4ad4009140 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4af4003980 con 0x7f4ad4009140 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:05:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 10 monclient: dump: 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2064348314 0 0) 0x7f4ad4009e80 con 0x7f4ad4009140 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.890+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ad400c420 con 0x7f4ad4009140 2024-09-16T03:05:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4291 (unknown 0 0 1465764260) 0x7f4af4003980 con 0x7f4ad4009140 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4af3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4af3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 >> v1:172.21.15.46:6789/0 conn(0x7f4ad4009140 legacy=0x7f4ad40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4af3fff640 1 -- 172.21.15.45:0/660245148 >> v1:172.21.15.46:6789/0 conn(0x7f4ad4009140 legacy=0x7f4ad40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 6, 2024-09-16T03:05:26.912 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 48699, 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:26.913 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:05:26.914 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.915 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:05:26.916 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:05:26.917 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:26.919 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:05:26.919 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:26.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.919 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:26.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.921 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:05:26.922 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.923 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.924 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:05:26.925 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:05:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 10 librados: watch_flush enter 2024-09-16T03:05:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 10 librados: watch_flush exit 2024-09-16T03:05:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 20 client.4475.objecter shutdown clearing up homeless session... 2024-09-16T03:05:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 10 client.4475.objecter successfully canceled tick 2024-09-16T03:05:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 >> v1:172.21.15.46:6812/149360280 conn(0x7f4ae003e000 legacy=0x7f4ae00404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 10 monclient: shutdown 2024-09-16T03:05:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:05:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 >> v1:172.21.15.45:6790/0 conn(0x7f4afc135950 legacy=0x7f4afc1cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:05:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4afb7fe640 1 -- 172.21.15.45:0/660245148 reap_dead start 2024-09-16T03:05:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4afb7fe640 1 -- 172.21.15.45:0/660245148 reap_dead start 2024-09-16T03:05:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 shutdown_connections 2024-09-16T03:05:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 >> 172.21.15.45:0/660245148 conn(0x7f4afc0a2bd0 msgr2=0x7f4afc13f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:05:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 shutdown_connections 2024-09-16T03:05:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 1 -- 172.21.15.45:0/660245148 wait complete. 2024-09-16T03:05:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:05:26.894+0000 7f4b01975640 1 librados: shutdown 2024-09-16T03:05:26.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:05:26.929 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:06:00.126 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 97% (12545/12889) Updating files: 98% (12632/12889) Updating files: 99% (12761/12889) Updating files: 100% (12889/12889) Updating files: 100% (12889/12889), done. 2024-09-16T03:06:00.270 INFO:tasks.workunit.client.0.smithi046.stderr:Note: switching to '7498493279982dfe87f00616198a5967475f6169'. 2024-09-16T03:06:00.270 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr: git switch -c 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr:Or undo this operation with: 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-16T03:06:00.271 INFO:tasks.workunit.client.0.smithi046.stderr: git switch - 2024-09-16T03:06:00.272 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-16T03:06:00.272 INFO:tasks.workunit.client.0.smithi046.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T03:06:00.272 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-16T03:06:00.272 INFO:tasks.workunit.client.0.smithi046.stderr:HEAD is now at 74984932799 Merge pull request #59797 from zdover23/wip-doc-2024-09-15-install-install-storage-cluster 2024-09-16T03:06:00.273 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T03:06:00.320 INFO:tasks.workunit.client.0.smithi046.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T03:06:00.323 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T03:06:00.324 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T03:06:01.125 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T03:06:01.234 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T03:06:01.293 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T03:06:01.295 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T03:06:01.295 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T03:06:01.341 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T03:06:01.347 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T03:06:01.347 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T03:06:01.395 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-16T03:06:01.396 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-16T03:06:01.396 DEBUG:teuthology.orchestra.run.smithi046:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-16T03:06:01.447 INFO:tasks.workunit.client.0.smithi046.stderr:+ parallel=1 2024-09-16T03:06:01.447 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' '' = --serial ']' 2024-09-16T03:06:01.447 INFO:tasks.workunit.client.0.smithi046.stderr:+ crimson=0 2024-09-16T03:06:01.447 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' '' = --crimson ']' 2024-09-16T03:06:01.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ color= 2024-09-16T03:06:01.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -t 1 ']' 2024-09-16T03:06:01.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-16T03:06:01.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T03:06:01.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T03:06:01.449 INFO:tasks.workunit.client.0.smithi046.stderr:+ declare -A pids 2024-09-16T03:06:01.449 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.449 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.449 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_aio 2024-09-16T03:06:01.449 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_aio' 2024-09-16T03:06:01.450 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_aio 2024-09-16T03:06:01.450 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.452 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_aio 2024-09-16T03:06:01.453 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28493 2024-09-16T03:06:01.453 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-16T03:06:01.453 INFO:tasks.workunit.client.0.smithi046.stdout:test api_aio on pid 28493 2024-09-16T03:06:01.454 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_aio on pid 28493' 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28493 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_aio_pp 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_aio_pp' 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_aio_pp 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-16T03:06:01.456 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-16T03:06:01.456 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_aio: /' 2024-09-16T03:06:01.456 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_aio_pp 2024-09-16T03:06:01.456 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28501 2024-09-16T03:06:01.457 INFO:tasks.workunit.client.0.smithi046.stdout:test api_aio_pp on pid 28501 2024-09-16T03:06:01.458 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_aio_pp on pid 28501' 2024-09-16T03:06:01.458 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28501 2024-09-16T03:06:01.458 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.458 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_io 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_io' 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_io 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-16T03:06:01.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-16T03:06:01.460 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-16T03:06:01.460 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_io 2024-09-16T03:06:01.460 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28509 2024-09-16T03:06:01.460 INFO:tasks.workunit.client.0.smithi046.stdout:test api_io on pid 28509 2024-09-16T03:06:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_io on pid 28509' 2024-09-16T03:06:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28509 2024-09-16T03:06:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_io_pp 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_io_pp' 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_io_pp 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_io.log 2024-09-16T03:06:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_io: /' 2024-09-16T03:06:01.464 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_io_pp 2024-09-16T03:06:01.464 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28517 2024-09-16T03:06:01.464 INFO:tasks.workunit.client.0.smithi046.stdout:test api_io_pp on pid 28517 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_io_pp on pid 28517' 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28517 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_asio 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_asio' 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.467 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.467 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_asio 2024-09-16T03:06:01.467 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.467 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-16T03:06:01.467 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-16T03:06:01.467 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-16T03:06:01.467 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_asio 2024-09-16T03:06:01.468 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28525 2024-09-16T03:06:01.468 INFO:tasks.workunit.client.0.smithi046.stdout:test api_asio on pid 28525 2024-09-16T03:06:01.469 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_asio on pid 28525' 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28525 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_list 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_list' 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_list 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.470 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.471 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.471 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.471 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-16T03:06:01.471 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-16T03:06:01.471 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_asio: /' 2024-09-16T03:06:01.472 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_list 2024-09-16T03:06:01.472 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28533 2024-09-16T03:06:01.473 INFO:tasks.workunit.client.0.smithi046.stdout:test api_list on pid 28533 2024-09-16T03:06:01.474 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_list on pid 28533' 2024-09-16T03:06:01.474 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28533 2024-09-16T03:06:01.474 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.474 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_lock 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_lock' 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_list.log 2024-09-16T03:06:01.475 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_list: /' 2024-09-16T03:06:01.476 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_lock 2024-09-16T03:06:01.477 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.480 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_lock 2024-09-16T03:06:01.481 INFO:tasks.workunit.client.0.smithi046.stdout:test api_lock on pid 28541 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28541 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_lock on pid 28541' 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28541 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-16T03:06:01.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-16T03:06:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_lock_pp 2024-09-16T03:06:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_lock_pp' 2024-09-16T03:06:01.495 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_lock: /' 2024-09-16T03:06:01.496 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_lock_pp 2024-09-16T03:06:01.496 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.499 INFO:tasks.workunit.client.0.smithi046.stdout:test api_lock_pp on pid 28560 2024-09-16T03:06:01.499 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_lock_pp 2024-09-16T03:06:01.499 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28560 2024-09-16T03:06:01.499 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_lock_pp on pid 28560' 2024-09-16T03:06:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28560 2024-09-16T03:06:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_misc 2024-09-16T03:06:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-16T03:06:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_misc' 2024-09-16T03:06:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_misc 2024-09-16T03:06:01.502 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.502 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.503 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.503 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.503 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-16T03:06:01.503 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-16T03:06:01.503 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-16T03:06:01.503 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.510 INFO:tasks.workunit.client.0.smithi046.stdout:test api_misc on pid 28620 2024-09-16T03:06:01.510 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_misc 2024-09-16T03:06:01.510 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28620 2024-09-16T03:06:01.511 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_misc on pid 28620' 2024-09-16T03:06:01.511 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28620 2024-09-16T03:06:01.511 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.511 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.511 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-16T03:06:01.511 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_misc_pp 2024-09-16T03:06:01.511 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_misc_pp' 2024-09-16T03:06:01.511 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_misc_pp 2024-09-16T03:06:01.512 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.512 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.512 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.512 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.512 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-16T03:06:01.513 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_misc: /' 2024-09-16T03:06:01.513 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-16T03:06:01.514 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.516 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_misc_pp 2024-09-16T03:06:01.516 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-16T03:06:01.516 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [----------] Global test environment set-up. 2024-09-16T03:06:01.516 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.503+0000 7f7b93692880 10 monclient: get_monmap_and_config 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.503+0000 7f7b93692880 10 monclient: build_initial_monmap 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.503+0000 7f7b93692880 10 monclient: monmap: 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: epoch 0 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: last_changed 0.000000 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: created 0.000000 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: election_strategy: 1 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.517 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 1 Processor -- start 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 1 -- start start 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 10 monclient: init 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 10 monclient: _reopen_session rank -1 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 10 monclient(hunting): picked mon.noname-b con 0x55ba50640300 addr v1:172.21.15.46:6789/0 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 10 monclient(hunting): picked mon.noname-a con 0x55ba506620a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 10 monclient(hunting): picked mon.noname-c con 0x55ba506624b0 addr v1:172.21.15.45:6790/0 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55ba506df750 con 0x55ba506620a0 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55ba50642b30 con 0x55ba50640300 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b9119c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x55ba50640300 0x55ba506de930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42432/0 (socket says 172.21.15.46:42432) 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b9119c640 1 -- 172.21.15.46:0/3160720017 learned_addr learned my addr 172.21.15.46:0/3160720017 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 1 -- 172.21.15.46:0/3160720017 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55ba506459e0 con 0x55ba506624b0 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 10 monclient(hunting): _renew_subs 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b93692880 10 monclient(hunting): authenticate will time out at 1045.229504s 2024-09-16T03:06:01.518 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.507+0000 7f7b9099b640 1 -- 172.21.15.46:0/3160720017 >> v1:172.21.15.45:6789/0 conn(0x55ba506620a0 legacy=0x55ba506df040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:06:01.519 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1920567522 0 0) 0x55ba506459e0 con 0x55ba506624b0 2024-09-16T03:06:01.519 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient(hunting): _negotiate 2024-09-16T03:06:01.519 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:06:01.519 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28635 2024-09-16T03:06:01.519 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10test api_misc_pp on pid 28635 2024-09-16T03:06:01.520 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_misc_pp on pid 28635' 2024-09-16T03:06:01.520 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28635 2024-09-16T03:06:01.520 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.520 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.520 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-16T03:06:01.520 INFO:tasks.workunit.client.0.smithi046.stdout: monclient(hunting): _init_auth creating new auth 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient(hunting): my global_id is 4499 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b74001e20 con 0x55ba506624b0 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705507047 0 0) 0x55ba50642b30 con 0x55ba50640300 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient(hunting): _negotiate 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient(hunting): my global_id is 4486 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55ba506459e0 con 0x55ba50640300 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1505176989 0 0) 0x7f7b74001e20 con 0x55ba506624b0 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 >> v1:172.21.15.46:6789/0 conn(0x55ba50640300 legacy=0x55ba506de930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 >> v1:172.21.15.45:6789/0 conn(0x55ba506620a0 legacy=0x55ba506df040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: _finish_hunting 0 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 monclient: found mon.noname-c 2024-09-16T03:06:01.521 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.45:6790/0 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba50645450 con 0x55ba506624b0 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: _finish_auth 0 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: _check_auth_tickets 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7b8c002a40 con 0x55ba506624b0 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: dump: 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: epoch 1 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:06:01.522 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:06:01.523 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: election_strategy: 1 2024-09-16T03:06:01.523 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:06:01.523 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:06:01.523 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:06:01.523 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_tier_pp 2024-09-16T03:06:01.523 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.524 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.524 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.524 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.524 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_tier_pp' 2024-09-16T03:06:01.524 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-16T03:06:01.524 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-16T03:06:01.524 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b93692880 5 monclient: authenticate success, global_id 4499 2024-09-16T03:06:01.524 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b93692880 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:06:01.524 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b8c002be0 con 0x55ba506624b0 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 1 -- 172.21.15.46:0/3160720017 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7b8c003070 con 0x55ba506624b0 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.511+0000 7f7b837fe640 10 monclient: dump: 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: epoch 1 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: election_strategy: 1 2024-09-16T03:06:01.525 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient: get_monmap_and_config success 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 4 set_mon_vals no callback set 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient: shutdown 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 -- 172.21.15.46:0/3160720017 >> v1:172.21.15.45:6790/0 conn(0x55ba506624b0 legacy=0x55ba5078b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 -- 172.21.15.46:0/3160720017 shutdown_connections 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 -- 172.21.15.46:0/3160720017 >> 172.21.15.46:0/3160720017 conn(0x55ba5063f810 msgr2=0x55ba50644d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 -- 172.21.15.46:0/3160720017 shutdown_connections 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 -- 172.21.15.46:0/3160720017 wait complete. 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient: build_initial_monmap 2024-09-16T03:06:01.526 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient: monmap: 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: epoch 0 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: last_changed 0.000000 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: created 0.000000 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: election_strategy: 1 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 librados: starting msgr at 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 librados: starting objecter 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 Processor -- start 2024-09-16T03:06:01.527 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 -- start start 2024-09-16T03:06:01.528 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-16T03:06:01.528 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_tier_pp 2024-09-16T03:06:01.528 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 librados: setting wanted keys 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 librados: calling monclient init 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient: init 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient: _reopen_session rank -1 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient(hunting): picked mon.noname-c con 0x55ba50640300 addr v1:172.21.15.45:6790/0 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient(hunting): picked mon.noname-a con 0x55ba506620a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient(hunting): picked mon.noname-b con 0x55ba506624b0 addr v1:172.21.15.46:6789/0 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55ba506e4990 con 0x55ba506620a0 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55ba506e4c30 con 0x55ba506624b0 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55ba506e4ed0 con 0x55ba50640300 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient(hunting): _renew_subs 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b93692880 10 monclient(hunting): authenticate will time out at 1045.239069s 2024-09-16T03:06:01.529 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b9099b640 1 -- >> v1:172.21.15.45:6789/0 conn(0x55ba506620a0 legacy=0x55ba506e20d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b9119c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x55ba50640300 0x55ba506e19c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.46:47850/0 (socket says 172.21.15.46:47850) 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.515+0000 7f7b9119c640 1 -- 172.21.15.46:0/1554543546 learned_addr learned my addr 172.21.15.46:0/1554543546 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 client.?.objecter ms_handle_connect 0x55ba50640300 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 client.?.objecter ms_handle_connect 0x55ba506624b0 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525569772 0 0) 0x55ba506e4ed0 con 0x55ba50640300 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient(hunting): _negotiate 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient(hunting): my global_id is 4508 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b74001a60 con 0x55ba50640300 2024-09-16T03:06:01.530 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1217337198 0 0) 0x7f7b74001a60 con 0x55ba50640300 2024-09-16T03:06:01.531 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.531 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x55ba506e4ed0 con 0x55ba50640300 2024-09-16T03:06:01.531 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7b840016a0 con 0x55ba50640300 2024-09-16T03:06:01.531 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:06:01.531 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:06:01.531 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient(hunting): dump: 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: epoch 1 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: election_strategy: 1 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1249746072 0 0) 0x55ba506e4ed0 con 0x55ba50640300 2024-09-16T03:06:01.532 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 >> v1:172.21.15.46:6789/0 conn(0x55ba506624b0 legacy=0x55ba50792a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 >> v1:172.21.15.45:6789/0 conn(0x55ba506620a0 legacy=0x55ba506e20d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient: _finish_hunting 0 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 monclient: found mon.c 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba506f3a40 con 0x55ba50640300 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient: _finish_auth 0 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient: _check_auth_tickets 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 5 monclient: authenticate success, global_id 4508 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 10 monclient: _renew_subs 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x55ba506e3a90 con 0x55ba50640300 2024-09-16T03:06:01.533 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:06:01.534 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 10 monclient: _renew_subs 2024-09-16T03:06:01.534 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:06:01.534 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.4test api_tier_pp on pid 28662 2024-09-16T03:06:01.534 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_tier_pp 2024-09-16T03:06:01.534 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28662 2024-09-16T03:06:01.534 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_tier_pp on pid 28662' 2024-09-16T03:06:01.534 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28662 2024-09-16T03:06:01.534 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_pool 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stdout:5:6790/0 -- mon_subscribe({osdmap=0}) -- 0x55ba506e27e0 con 0x55ba50640300 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 1 librados: init done 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b93692880 10 librados: wait_for_osdmap waiting 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b84001d00 con 0x55ba50640300 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.519+0000 7f7b837fe640 4 set_mon_vals no callback set 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7b840036a0 con 0x55ba50640300 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:06:01.535 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 10 monclient: dump: 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: epoch 1 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: election_strategy: 1 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7b84003900 con 0x55ba50640300 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 4102739278 0 0) 0x7f7b84049bd0 con 0x55ba50640300 2024-09-16T03:06:01.536 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b9099b640 10 monclient: get_auth_request con 0x7f7b7403c1b0 auth_method 0 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_map(20..20 src has 1..20) 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 3 client.4508.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 3 client.4508.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 20 client.4508.objecter dump_active .. 0 homeless 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b93692880 10 librados: wait_for_osdmap done waiting 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b93692880 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7b840018c0 con 0x55ba50640300 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 10 client.4508.objecter ms_handle_connect 0x7f7b7403c1b0 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7f7b84014dd0 con 0x55ba50640300 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b827fc640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-16T03:06:01.537 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b82ffd640 10 client.4508.objecter _get_latest_version latest 20, have it 2024-09-16T03:06:01.538 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_pool' 2024-09-16T03:06:01.538 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_pool 2024-09-16T03:06:01.538 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.538 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_pool 2024-09-16T03:06:01.538 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.52test api_pool on pid 28669 2024-09-16T03:06:01.540 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28669 2024-09-16T03:06:01.540 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_pool on pid 28669' 2024-09-16T03:06:01.540 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28669 2024-09-16T03:06:01.540 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.540 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_snapshots 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_snapshots' 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-09-16T03:06:01.542 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-16T03:06:01.542 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-16T03:06:01.542 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_snapshots 2024-09-16T03:06:01.542 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.542 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_pool: /' 2024-09-16T03:06:01.542 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-16T03:06:01.542 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_snapshots 2024-09-16T03:06:01.543 INFO:tasks.workunit.client.0.smithi046.stdout:test api_snapshots on pid 28696 2024-09-16T03:06:01.543 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28696 2024-09-16T03:06:01.543 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_snapshots on pid 28696' 2024-09-16T03:06:01.543 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28696 2024-09-16T03:06:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-16T03:06:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-16T03:06:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_snapshots_pp 2024-09-16T03:06:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_snapshots_pp' 2024-09-16T03:06:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-16T03:06:01.550 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-16T03:06:01.552 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_snapshots_pp 2024-09-16T03:06:01.552 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.555 INFO:tasks.workunit.client.0.smithi046.stdout:test api_snapshots_pp on pid 28734 2024-09-16T03:06:01.555 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_snapshots_pp 2024-09-16T03:06:01.555 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28734 2024-09-16T03:06:01.555 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_snapshots_pp on pid 28734' 2024-09-16T03:06:01.555 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28734 2024-09-16T03:06:01.555 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_stat 2024-09-16T03:06:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_stat' 2024-09-16T03:06:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_stat 2024-09-16T03:06:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.558 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-16T03:06:01.559 INFO:tasks.workunit.client.0.smithi046.stdout:test api_stat on pid 28748 2024-09-16T03:06:01.560 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_stat 2024-09-16T03:06:01.560 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28748 2024-09-16T03:06:01.560 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_stat on pid 28748' 2024-09-16T03:06:01.560 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28748 2024-09-16T03:06:01.560 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_stat_pp 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_stat_pp' 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-16T03:06:01.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-16T03:06:01.562 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.562 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-16T03:06:01.563 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.563 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.563 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.563 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-16T03:06:01.564 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-16T03:06:01.564 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_stat: /' 2024-09-16T03:06:01.578 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_stat_pp 2024-09-16T03:06:01.578 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.581 INFO:tasks.workunit.client.0.smithi046.stdout:test api_stat_pp on pid 28780 2024-09-16T03:06:01.581 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_stat_pp 2024-09-16T03:06:01.581 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28780 2024-09-16T03:06:01.581 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_stat_pp on pid 28780' 2024-09-16T03:06:01.581 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28780 2024-09-16T03:06:01.581 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.581 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.581 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-16T03:06:01.582 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.582 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.583 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.583 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.583 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-16T03:06:01.583 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-16T03:06:01.585 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_watch_notify 2024-09-16T03:06:01.585 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-16T03:06:01.585 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_watch_notify' 2024-09-16T03:06:01.593 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.595 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_watch_notify 2024-09-16T03:06:01.597 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_watch_notify 2024-09-16T03:06:01.598 INFO:tasks.workunit.client.0.smithi046.stdout:test api_watch_notify on pid 28828 2024-09-16T03:06:01.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28828 2024-09-16T03:06:01.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_watch_notify on pid 28828' 2024-09-16T03:06:01.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28828 2024-09-16T03:06:01.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-16T03:06:01.602 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.602 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.602 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.602 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.602 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-16T03:06:01.603 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-16T03:06:01.605 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_watch_notify_pp 2024-09-16T03:06:01.605 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_watch_notify_pp' 2024-09-16T03:06:01.607 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-16T03:06:01.612 INFO:tasks.workunit.client.0.smithi046.stdout:3+0000 7f7b93692880 10 client.4508.objecter create_pool name=ceph_test_rados_api_asio 2024-09-16T03:06:01.612 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b93692880 10 client.4508.objecter pool_op_submit 1 2024-09-16T03:06:01.612 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b93692880 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:06:01.612 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.523+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6790/0 -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x55ba505ee880 con 0x55ba50640300 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 10 ==== pool_op_reply(tid 1 (0) Success v21) ==== 43+0+0 (unknown 838695312 0 0) 0x7f7b84015700 con 0x55ba50640300 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 pool_op_reply(tid 1 (0) Success v21) 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 10 client.4508.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v21) 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 10 client.4508.objecter have request 1 at 0x55ba50646040 Op: create 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 20 client.4508.objecter waiting for client to reach epoch 21 before calling back 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 10 monclient: _renew_subs 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=21}) -- 0x55ba506e4ed0 con 0x55ba50640300 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.603+0000 7f7b827fc640 10 client.4508.objecter done 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== mon.2 v1:172.21.15.45:6790/0 11 ==== osd_map(21..21 src has 1..21) ==== 3224+0+0 (unknown 1565691360 0 0) 0x7f7b84048a30 con 0x55ba50640300 2024-09-16T03:06:01.613 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b827fc640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_map(21..21 src has 1..21) 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b827fc640 3 client.4508.objecter handle_osd_map got epochs [21,21] > 20 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b827fc640 3 client.4508.objecter handle_osd_map decoding incremental epoch 21 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b827fc640 20 client.4508.objecter dump_active .. 0 homeless 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 10 librados: set snap read head -> 1 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 10 librados: writefull oid=exist nspace= 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50632e90 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 2 osd.1 2024-09-16T03:06:01.614 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:01.615 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_watch_notify_pp 2024-09-16T03:06:01.615 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.625 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_watch_notify_pp 2024-09-16T03:06:01.626 INFO:tasks.workunit.client.0.smithi046.stdout: test api_watch_notify_pp on pid 28895 2024-09-16T03:06:01.626 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28895 2024-09-16T03:06:01.626 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_watch_notify_pp on pid 28895' 2024-09-16T03:06:01.626 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28895 2024-09-16T03:06:01.626 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.626 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.626 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-16T03:06:01.627 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.627 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.627 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.627 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.628 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-16T03:06:01.630 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_cmd 2024-09-16T03:06:01.632 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_cmd' 2024-09-16T03:06:01.632 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-16T03:06:01.632 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-16T03:06:01.641 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_cmd 2024-09-16T03:06:01.641 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.644 INFO:tasks.workunit.client.0.smithi046.stdout:test api_cmd on pid 28930 2024-09-16T03:06:01.645 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_cmd 2024-09-16T03:06:01.645 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28930 2024-09-16T03:06:01.645 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_cmd on pid 28930' 2024-09-16T03:06:01.645 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28930 2024-09-16T03:06:01.645 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.645 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.645 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_cmd_pp 2024-09-16T03:06:01.645 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_cmd_pp' 2024-09-16T03:06:01.645 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-16T03:06:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_cmd_pp 2024-09-16T03:06:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-16T03:06:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_cmd: /' 2024-09-16T03:06:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-16T03:06:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.650 INFO:tasks.workunit.client.0.smithi046.stdout:test api_cmd_pp on pid 28944 2024-09-16T03:06:01.650 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_cmd_pp 2024-09-16T03:06:01.650 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28944 2024-09-16T03:06:01.650 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_cmd_pp on pid 28944' 2024-09-16T03:06:01.650 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28944 2024-09-16T03:06:01.650 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.650 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.651 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_service 2024-09-16T03:06:01.651 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-16T03:06:01.651 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_service' 2024-09-16T03:06:01.652 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_service 2024-09-16T03:06:01.652 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.652 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.652 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.652 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.652 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.654 INFO:tasks.workunit.client.0.smithi046.stdout:test api_service on pid 28952 2024-09-16T03:06:01.654 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_service 2024-09-16T03:06:01.654 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28952 2024-09-16T03:06:01.654 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_service on pid 28952' 2024-09-16T03:06:01.654 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28952 2024-09-16T03:06:01.654 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.654 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.655 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-16T03:06:01.655 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-16T03:06:01.656 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-16T03:06:01.658 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.658 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_service_pp 2024-09-16T03:06:01.658 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.658 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.658 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.658 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_service_pp' 2024-09-16T03:06:01.659 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_service: /' 2024-09-16T03:06:01.659 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_service_pp 2024-09-16T03:06:01.659 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-16T03:06:01.659 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_service.log 2024-09-16T03:06:01.659 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-16T03:06:01.660 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.662 INFO:tasks.workunit.client.0.smithi046.stdout:test api_service_pp on pid 28970 2024-09-16T03:06:01.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_service_pp 2024-09-16T03:06:01.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28970 2024-09-16T03:06:01.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_service_pp on pid 28970' 2024-09-16T03:06:01.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28970 2024-09-16T03:06:01.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-16T03:06:01.664 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.665 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.665 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.665 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.665 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-16T03:06:01.665 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-16T03:06:01.665 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-16T03:06:01.670 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_c_write_operations 2024-09-16T03:06:01.670 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_c_write_operations' 2024-09-16T03:06:01.670 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_c_write_operations 2024-09-16T03:06:01.670 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.673 INFO:tasks.workunit.client.0.smithi046.stdout:test api_c_write_operations on pid 28989 2024-09-16T03:06:01.673 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_c_write_operations 2024-09-16T03:06:01.674 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28989 2024-09-16T03:06:01.674 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_c_write_operations on pid 28989' 2024-09-16T03:06:01.674 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=28989 2024-09-16T03:06:01.674 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.674 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.675 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s api_c_read_operations 2024-09-16T03:06:01.675 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' api_c_read_operations' 2024-09-16T03:06:01.675 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-16T03:06:01.676 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo api_c_read_operations 2024-09-16T03:06:01.677 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.677 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.677 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.677 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.677 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.678 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-16T03:06:01.680 INFO:tasks.workunit.client.0.smithi046.stdout:test api_c_read_operations on pid 29006 2024-09-16T03:06:01.680 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=api_c_read_operations 2024-09-16T03:06:01.680 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29006 2024-09-16T03:06:01.680 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test api_c_read_operations on pid 29006' 2024-09-16T03:06:01.680 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29006 2024-09-16T03:06:01.680 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.680 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.680 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-16T03:06:01.684 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s list_parallel 2024-09-16T03:06:01.684 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' list_parallel' 2024-09-16T03:06:01.684 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.684 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.684 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.684 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.685 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-16T03:06:01.685 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-16T03:06:01.685 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-16T03:06:01.685 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-16T03:06:01.685 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-16T03:06:01.686 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.701 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo list_parallel 2024-09-16T03:06:01.703 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=list_parallel 2024-09-16T03:06:01.703 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29053 2024-09-16T03:06:01.704 INFO:tasks.workunit.client.0.smithi046.stdout:test list_parallel on pid 29053 2024-09-16T03:06:01.705 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test list_parallel on pid 29053' 2024-09-16T03:06:01.705 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29053 2024-09-16T03:06:01.705 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.705 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-16T03:06:01.705 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.705 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s open_pools_parallel 2024-09-16T03:06:01.706 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' open_pools_parallel' 2024-09-16T03:06:01.706 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo open_pools_parallel 2024-09-16T03:06:01.706 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.706 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.706 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.706 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.706 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.706 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-16T03:06:01.707 INFO:tasks.workunit.client.0.smithi046.stdout:test open_pools_parallel on pid 29069 2024-09-16T03:06:01.707 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=open_pools_parallel 2024-09-16T03:06:01.707 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29069 2024-09-16T03:06:01.707 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test open_pools_parallel on pid 29069' 2024-09-16T03:06:01.707 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29069 2024-09-16T03:06:01.707 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:06:01.707 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.708 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s delete_pools_parallel 2024-09-16T03:06:01.708 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' delete_pools_parallel' 2024-09-16T03:06:01.708 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-16T03:06:01.708 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-16T03:06:01.708 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo delete_pools_parallel 2024-09-16T03:06:01.709 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.709 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.709 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.710 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.710 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-16T03:06:01.710 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-16T03:06:01.714 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-16T03:06:01.717 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ list_parallel: /' 2024-09-16T03:06:01.718 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.721 INFO:tasks.workunit.client.0.smithi046.stdout:test delete_pools_parallel on pid 29082 2024-09-16T03:06:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=delete_pools_parallel 2024-09-16T03:06:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29082 2024-09-16T03:06:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test delete_pools_parallel on pid 29082' 2024-09-16T03:06:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29082 2024-09-16T03:06:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.722 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.722 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-16T03:06:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.724 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-16T03:06:01.724 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-16T03:06:01.725 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s cls 2024-09-16T03:06:01.725 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' cls' 2024-09-16T03:06:01.726 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo cls 2024-09-16T03:06:01.728 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.729 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-16T03:06:01.731 INFO:tasks.workunit.client.0.smithi046.stdout:test cls on pid 29118 2024-09-16T03:06:01.731 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=cls 2024-09-16T03:06:01.731 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29118 2024-09-16T03:06:01.731 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test cls on pid 29118' 2024-09-16T03:06:01.731 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29118 2024-09-16T03:06:01.731 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.732 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.733 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-16T03:06:01.735 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.735 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.735 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.735 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.738 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s cmd 2024-09-16T03:06:01.738 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' cmd' 2024-09-16T03:06:01.741 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_cls.log 2024-09-16T03:06:01.750 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo cmd 2024-09-16T03:06:01.750 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.754 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_cls 2024-09-16T03:06:01.755 INFO:tasks.workunit.client.0.smithi046.stdout:test cmd on pid 29181 2024-09-16T03:06:01.755 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=cmd 2024-09-16T03:06:01.756 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29181 2024-09-16T03:06:01.756 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test cmd on pid 29181' 2024-09-16T03:06:01.756 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29181 2024-09-16T03:06:01.756 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.756 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.756 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ cls: /' 2024-09-16T03:06:01.757 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s handler_error 2024-09-16T03:06:01.757 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' handler_error' 2024-09-16T03:06:01.757 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-16T03:06:01.759 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.759 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.759 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.759 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.760 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-16T03:06:01.760 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_cmd 2024-09-16T03:06:01.761 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ cmd: /' 2024-09-16T03:06:01.777 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.779 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo handler_error 2024-09-16T03:06:01.781 INFO:tasks.workunit.client.0.smithi046.stdout:test handler_error on pid 29253 2024-09-16T03:06:01.781 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=handler_error 2024-09-16T03:06:01.782 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29253 2024-09-16T03:06:01.782 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test handler_error on pid 29253' 2024-09-16T03:06:01.782 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29253 2024-09-16T03:06:01.782 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.782 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.782 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-16T03:06:01.783 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.783 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.783 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.783 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.783 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s io 2024-09-16T03:06:01.784 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' io' 2024-09-16T03:06:01.784 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.785 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ handler_error: /' 2024-09-16T03:06:01.785 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-16T03:06:01.786 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_handler_error 2024-09-16T03:06:01.788 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo io 2024-09-16T03:06:01.789 INFO:tasks.workunit.client.0.smithi046.stdout:test io on pid 29285 2024-09-16T03:06:01.790 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=io 2024-09-16T03:06:01.790 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29285 2024-09-16T03:06:01.790 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test io on pid 29285' 2024-09-16T03:06:01.790 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29285 2024-09-16T03:06:01.790 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.790 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.791 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-16T03:06:01.793 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.793 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.793 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.793 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.793 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_io 2024-09-16T03:06:01.794 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_io.log 2024-09-16T03:06:01.794 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ io: /' 2024-09-16T03:06:01.795 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s ec_io 2024-09-16T03:06:01.795 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' ec_io' 2024-09-16T03:06:01.796 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo ec_io 2024-09-16T03:06:01.796 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.800 INFO:tasks.workunit.client.0.smithi046.stdout:test ec_io on pid 29326 2024-09-16T03:06:01.800 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=ec_io 2024-09-16T03:06:01.800 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29326 2024-09-16T03:06:01.800 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test ec_io on pid 29326' 2024-09-16T03:06:01.800 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29326 2024-09-16T03:06:01.800 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.800 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.801 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-16T03:06:01.801 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s list 2024-09-16T03:06:01.801 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' list' 2024-09-16T03:06:01.801 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.802 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.802 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.802 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo list 2024-09-16T03:06:01.802 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.802 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.802 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ ec_io: /' 2024-09-16T03:06:01.802 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_ec_io 2024-09-16T03:06:01.803 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-16T03:06:01.804 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=list 2024-09-16T03:06:01.804 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29362 2024-09-16T03:06:01.804 INFO:tasks.workunit.client.0.smithi046.stdout:test list on pid 29362 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test list on pid 29362' 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29362 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s ec_list 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' ec_list' 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo ec_list 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.807 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_list 2024-09-16T03:06:01.807 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_list.log 2024-09-16T03:06:01.811 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.817 INFO:tasks.workunit.client.0.smithi046.stdout:test ec_list on pid 29389 2024-09-16T03:06:01.817 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=ec_list 2024-09-16T03:06:01.817 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29389 2024-09-16T03:06:01.817 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test ec_list on pid 29389' 2024-09-16T03:06:01.817 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29389 2024-09-16T03:06:01.817 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-16T03:06:01.817 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.817 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.818 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s misc 2024-09-16T03:06:01.818 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' misc' 2024-09-16T03:06:01.818 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.818 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.819 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.819 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.819 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.819 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_ec_list 2024-09-16T03:06:01.820 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ list: /' 2024-09-16T03:06:01.830 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ ec_list: /' 2024-09-16T03:06:01.830 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-16T03:06:01.832 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo misc 2024-09-16T03:06:01.833 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=misc 2024-09-16T03:06:01.833 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29434 2024-09-16T03:06:01.833 INFO:tasks.workunit.client.0.smithi046.stdout:test misc on pid 29434 2024-09-16T03:06:01.834 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test misc on pid 29434' 2024-09-16T03:06:01.834 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29434 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s pool 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' pool' 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_misc 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_misc.log 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo pool 2024-09-16T03:06:01.835 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ misc: /' 2024-09-16T03:06:01.837 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.840 INFO:tasks.workunit.client.0.smithi046.stdout:test pool on pid 29469 2024-09-16T03:06:01.841 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=pool 2024-09-16T03:06:01.841 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29469 2024-09-16T03:06:01.841 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test pool on pid 29469' 2024-09-16T03:06:01.841 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29469 2024-09-16T03:06:01.841 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.841 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-16T03:06:01.841 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.841 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s read_operations 2024-09-16T03:06:01.842 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.842 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.842 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.842 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.842 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_pool 2024-09-16T03:06:01.842 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' read_operations' 2024-09-16T03:06:01.842 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ pool: /' 2024-09-16T03:06:01.843 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo read_operations 2024-09-16T03:06:01.844 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.846 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_pool.log 2024-09-16T03:06:01.850 INFO:tasks.workunit.client.0.smithi046.stdout:test read_operations on pid 29501 2024-09-16T03:06:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=read_operations 2024-09-16T03:06:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29501 2024-09-16T03:06:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test read_operations on pid 29501' 2024-09-16T03:06:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29501 2024-09-16T03:06:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-16T03:06:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_read_operations 2024-09-16T03:06:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-16T03:06:01.853 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ read_operations: /' 2024-09-16T03:06:01.853 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s snapshots 2024-09-16T03:06:01.853 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' snapshots' 2024-09-16T03:06:01.854 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo snapshots 2024-09-16T03:06:01.854 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.856 INFO:tasks.workunit.client.0.smithi046.stdout:test snapshots on pid 29517 2024-09-16T03:06:01.856 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=snapshots 2024-09-16T03:06:01.856 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29517 2024-09-16T03:06:01.857 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test snapshots on pid 29517' 2024-09-16T03:06:01.857 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29517 2024-09-16T03:06:01.857 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.857 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.857 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-16T03:06:01.857 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s watch_notify 2024-09-16T03:06:01.857 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' watch_notify' 2024-09-16T03:06:01.857 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo watch_notify 2024-09-16T03:06:01.857 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.858 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.858 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.858 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.858 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.858 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_snapshots 2024-09-16T03:06:01.858 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-16T03:06:01.859 INFO:tasks.workunit.client.0.smithi046.stdout:test watch_notify on pid 29525 2024-09-16T03:06:01.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=watch_notify 2024-09-16T03:06:01.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29525 2024-09-16T03:06:01.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test watch_notify on pid 29525' 2024-09-16T03:06:01.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29525 2024-09-16T03:06:01.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:06:01.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-16T03:06:01.860 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.860 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.860 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.861 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.861 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_watch_notify 2024-09-16T03:06:01.861 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-16T03:06:01.861 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ watch_notify: /' 2024-09-16T03:06:01.861 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ snapshots: /' 2024-09-16T03:06:01.861 INFO:tasks.workunit.client.0.smithi046.stderr:++ printf %25s write_operations 2024-09-16T03:06:01.862 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=' write_operations' 2024-09-16T03:06:01.862 INFO:tasks.workunit.client.0.smithi046.stderr:++ echo write_operations 2024-09-16T03:06:01.862 INFO:tasks.workunit.client.0.smithi046.stderr:++ awk '{print $1}' 2024-09-16T03:06:01.864 INFO:tasks.workunit.client.0.smithi046.stderr:+ ff=write_operations 2024-09-16T03:06:01.865 INFO:tasks.workunit.client.0.smithi046.stdout:test write_operations on pid 29539 2024-09-16T03:06:01.866 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29539 2024-09-16T03:06:01.867 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 'test write_operations on pid 29539' 2024-09-16T03:06:01.867 INFO:tasks.workunit.client.0.smithi046.stderr:+ pids[$f]=29539 2024-09-16T03:06:01.867 INFO:tasks.workunit.client.0.smithi046.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-16T03:06:01.867 INFO:tasks.workunit.client.0.smithi046.stderr:+ ret=0 2024-09-16T03:06:01.867 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:06:01.868 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:06:01.868 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28662 2024-09-16T03:06:01.868 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28662 2024-09-16T03:06:01.868 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -z '' ']' 2024-09-16T03:06:01.868 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:+ case $- in 2024-09-16T03:06:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:+ return 2024-09-16T03:06:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:+ ceph_test_neorados_write_operations 2024-09-16T03:06:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:+ sed 's/^/ write_operations: /' 2024-09-16T03:06:01.870 INFO:tasks.workunit.client.0.smithi046.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-16T03:06:02.732 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 2 2024-09-16T03:06:02.732 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 15 client.4508.objecter _send_op 2 to 6.1 on osd.1 2024-09-16T03:06:02.732 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:2 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x55ba50637360 con 0x55ba50633850 2024-09-16T03:06:02.732 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.732 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.607+0000 7f7b9199d640 10 monclient: get_auth_request con 0x55ba50633850 auth_method 0 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:01.611+0000 7f7b827fc640 10 client.4508.objecter ms_handle_connect 0x55ba50633850 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.683+0000 7f7b827fc640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 1 ==== osd_map(22..22 src has 1..22) ==== 13341+0+0 (unknown 3301203778 0 0) 0x7f7b8c0038b0 con 0x55ba50633850 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.683+0000 7f7b827fc640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_map(22..22 src has 1..22) 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.683+0000 7f7b827fc640 3 client.4508.objecter handle_osd_map got epochs [22,22] > 21 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.683+0000 7f7b827fc640 3 client.4508.objecter handle_osd_map decoding incremental epoch 22 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.683+0000 7f7b827fc640 10 client.4508.objecter checking op 2 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.683+0000 7f7b827fc640 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.683+0000 7f7b827fc640 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.683+0000 7f7b827fc640 20 client.4508.objecter dump_active .. 0 homeless 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.683+0000 7f7b827fc640 20 client.4508.objecter 2 6.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-09-16T03:06:02.733 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 2 ==== osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) ==== 149+0+0 (unknown 2278174585 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 2 ondisk uv 1 in 6.1 attempt 0 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 2 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 15 client.4508.objecter _finish_op 2 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 2 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-16T03:06:02.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 10 librados: Objecter returned from writefull r=0 2024-09-16T03:06:02.749 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50631e80 2024-09-16T03:06:02.749 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.749 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.749 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 3 osd.1 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 3 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 15 client.4508.objecter _send_op 3 to 6.1 on osd.1 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:3 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba50632330 con 0x55ba50633850 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50632710 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T03:06:02.750 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.751 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.751 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 10 client.4508.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 4 osd.1 2024-09-16T03:06:02.751 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.751 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 4 2024-09-16T03:06:02.751 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 15 client.4508.objecter _send_op 4 to 6.1 on osd.1 2024-09-16T03:06:02.751 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:4 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba50632bc0 con 0x55ba50633850 2024-09-16T03:06:02.751 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.751 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.727+0000 7f7b93692880 5 client.4508.objecter 2 in flight 2024-09-16T03:06:02.751 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 4165629867 0 3741568377) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 3 ondisk uv 1 in 6.1 attempt 0 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 5 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 3 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 15 client.4508.objecter _finish_op 3 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 3 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 1871807346 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.752 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 4 ondisk uv 0 in 6.1 attempt 0 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 10 client.4508.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 4 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 15 client.4508.objecter _finish_op 4 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 4 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (17 ms) 2024-09-16T03:06:02.753 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-16T03:06:02.754 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50631e80 2024-09-16T03:06:02.754 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.754 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.754 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.754 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.754 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 5 osd.1 2024-09-16T03:06:02.754 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.754 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 5 2024-09-16T03:06:02.754 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 15 client.4508.objecter _send_op 5 to 6.1 on osd.1 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:5 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba506326f0 con 0x55ba50633850 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 10 client.4508.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 6 osd.1 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.755 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 6 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 15 client.4508.objecter _send_op 6 to 6.1 on osd.1 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:6 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba50637bb0 con 0x55ba50633850 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.743+0000 7f7b93692880 5 client.4508.objecter 2 in flight 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 4165629867 0 3741568377) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 5 ondisk uv 1 in 6.1 attempt 0 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 5 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 5 2024-09-16T03:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 15 client.4508.objecter _finish_op 5 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 5 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 1871807346 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 6 ondisk uv 0 in 6.1 attempt 0 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 10 client.4508.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.757 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 6 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 15 client.4508.objecter _finish_op 6 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 6 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (4 ms) 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.758 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 7 osd.1 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 7 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 15 client.4508.objecter _send_op 7 to 6.1 on osd.1 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:7 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba50631e80 con 0x55ba50633850 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637fd0 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T03:06:02.759 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.760 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.760 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 10 client.4508.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 8 osd.1 2024-09-16T03:06:02.760 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.760 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 8 2024-09-16T03:06:02.771 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 15 client.4508.objecter _send_op 8 to 6.1 on osd.1 2024-09-16T03:06:02.771 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:8 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba50638480 con 0x55ba50633850 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.747+0000 7f7b93692880 5 client.4508.objecter 2 in flight 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 4165629867 0 3741568377) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 7 ondisk uv 1 in 6.1 attempt 0 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 5 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 7 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 15 client.4508.objecter _finish_op 7 2024-09-16T03:06:02.772 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 7 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.755+0000 7f7b9199d640 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 1871807346 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 8 ondisk uv 0 in 6.1 attempt 0 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 10 client.4508.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (18 ms) 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.773 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 8 2024-09-16T03:06:02.774 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-16T03:06:02.774 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 15 client.4508.objecter _finish_op 8 2024-09-16T03:06:02.774 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.774 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 8 2024-09-16T03:06:02.774 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.774 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T03:06:02.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 20 librados: queue_aio_write 0x55ba505b3530 completion 0x55ba50637fd0 write_seq 1 2024-09-16T03:06:02.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:06:02.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 9 osd.1 2024-09-16T03:06:02.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 9 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 15 client.4508.objecter _send_op 9 to 6.1 on osd.1 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:9 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x55ba50639830 con 0x55ba50633850 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.767+0000 7f7b93692880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 9 ==== osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) ==== 149+0+0 (unknown 2273072529 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 9 ondisk uv 2 in 6.1 attempt 0 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 9 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 15 client.4508.objecter _finish_op 9 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 9 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b9199d640 20 librados: complete_aio_write 0x55ba50637fd0 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (40 ms) 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 20 librados: queue_aio_write 0x55ba505b3530 completion 0x55ba50639bf0 write_seq 2 2024-09-16T03:06:02.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:06:02.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 10 osd.1 2024-09-16T03:06:02.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 10 2024-09-16T03:06:02.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 15 client.4508.objecter _send_op 10 to 6.1 on osd.1 2024-09-16T03:06:02.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:10 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x55ba50639830 con 0x55ba50633850 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.807+0000 7f7b93692880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 10 ==== osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) ==== 149+0+0 (unknown 4093439557 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 10 ondisk uv 3 in 6.1 attempt 0 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 10 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 15 client.4508.objecter _finish_op 10 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 10 2024-09-16T03:06:02.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b9199d640 20 librados: complete_aio_write 0x55ba50639bf0 2024-09-16T03:06:02.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (13 ms) 2024-09-16T03:06:02.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-16T03:06:02.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T03:06:02.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 20 librados: queue_aio_write 0x55ba505b3530 completion 0x55ba50639a20 write_seq 3 2024-09-16T03:06:02.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:06:02.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.832 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.833 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.833 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 11 osd.1 2024-09-16T03:06:02.833 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.833 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 11 2024-09-16T03:06:02.833 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 15 client.4508.objecter _send_op 11 to 6.1 on osd.1 2024-09-16T03:06:02.833 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:11 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x55ba507d5580 con 0x55ba50633850 2024-09-16T03:06:02.833 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.833 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.833 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.819+0000 7f7b93692880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 11 ==== osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) ==== 149+0+0 (unknown 3113466776 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 11 ondisk uv 4 in 6.1 attempt 0 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 11 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 15 client.4508.objecter _finish_op 11 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 11 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (9 ms) 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b9199d640 20 librados: complete_aio_write 0x55ba50639a20 2024-09-16T03:06:02.834 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.835 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.835 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.835 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.835 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.835 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 12 osd.1 2024-09-16T03:06:02.835 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.835 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 12 2024-09-16T03:06:02.835 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 15 client.4508.objecter _send_op 12 to 6.1 on osd.1 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:12 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba50638820 con 0x55ba50633850 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50638cf0 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 10 client.4508.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 13 osd.1 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 13 2024-09-16T03:06:02.836 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 15 client.4508.objecter _send_op 13 to 6.1 on osd.1 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:13 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba506391a0 con 0x55ba50633850 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.827+0000 7f7b93692880 5 client.4508.objecter 2 in flight 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 2316866906 0 3741568377) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 12 ondisk uv 4 in 6.1 attempt 0 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 5 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 12 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _finish_op 12 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 12 2024-09-16T03:06:02.837 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 35241290 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 13 ondisk uv 0 in 6.1 attempt 0 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 13 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _finish_op 13 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 13 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (3 ms) 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-16T03:06:02.838 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 14 osd.1 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 14 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _send_op 14 to 6.1 on osd.1 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:14 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba50638820 con 0x55ba50633850 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.839 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50638de0 2024-09-16T03:06:02.840 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.840 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 15 osd.1 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 15 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _send_op 15 to 6.1 on osd.1 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:15 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba50795560 con 0x55ba50633850 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 5 client.4508.objecter 2 in flight 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 2316866906 0 3741568377) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T03:06:02.841 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 14 ondisk uv 4 in 6.1 attempt 0 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 5 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 14 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _finish_op 14 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 14 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 35241290 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 15 ondisk uv 0 in 6.1 attempt 0 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.842 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 15 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _finish_op 15 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 15 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 16 osd.1 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.843 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 16 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _send_op 16 to 6.1 on osd.1 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:16 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba50795560 con 0x55ba50633850 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50638de0 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 17 osd.1 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.844 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 17 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _send_op 17 to 6.1 on osd.1 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:17 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x55ba507b59e0 con 0x55ba50633850 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 5 client.4508.objecter 2 in flight 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 2316866906 0 3741568377) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 16 ondisk uv 4 in 6.1 attempt 0 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 5 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 16 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _finish_op 16 2024-09-16T03:06:02.845 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 5 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 16 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 35241290 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 17 ondisk uv 0 in 6.1 attempt 0 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 17 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _finish_op 17 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-16T03:06:02.846 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 17 2024-09-16T03:06:02.847 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.847 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 librados: queue_aio_write 0x55ba505b3530 completion 0x55ba50638820 write_seq 4 2024-09-16T03:06:02.847 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.847 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:06:02.847 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.847 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.847 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.847 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 18 osd.1 2024-09-16T03:06:02.848 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.848 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 18 2024-09-16T03:06:02.848 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 15 client.4508.objecter _send_op 18 to 6.1 on osd.1 2024-09-16T03:06:02.848 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:18 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x55ba50795560 con 0x55ba50633850 2024-09-16T03:06:02.848 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.848 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.831+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.848 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 18 ==== osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) ==== 149+0+0 (unknown 1344583384 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.848 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) 2024-09-16T03:06:02.849 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.849 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 18 ondisk uv 5 in 6.1 attempt 0 2024-09-16T03:06:02.849 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.849 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 18 2024-09-16T03:06:02.849 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 15 client.4508.objecter _finish_op 18 2024-09-16T03:06:02.849 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (5 ms) 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 18 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 20 librados: complete_aio_write 0x55ba50638820 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 librados: queue_aio_write 0x55ba505b3530 completion 0x55ba50638a40 write_seq 5 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.850 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 19 osd.1 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 19 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 15 client.4508.objecter _send_op 19 to 6.1 on osd.1 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:19 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x55ba50795560 con 0x55ba50633850 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 19 ==== osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) ==== 149+0+0 (unknown 3450174116 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 19 ondisk uv 6 in 6.1 attempt 0 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 19 2024-09-16T03:06:02.851 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 15 client.4508.objecter _finish_op 19 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 19 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b9199d640 20 librados: complete_aio_write 0x55ba50638a40 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2 ms) 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 librados: queue_aio_write 0x55ba505b3530 completion 0x55ba50638de0 write_seq 6 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 10 client.4508.objecter _op_submit op 0x55ba50637700 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T03:06:02.852 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 10 client.4508.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-16T03:06:02.853 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter _get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.853 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 10 client.4508.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 20 osd.1 2024-09-16T03:06:02.853 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter get_session s=0x55ba506e2c90 osd=1 3 2024-09-16T03:06:02.853 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 15 client.4508.objecter _session_op_assign 1 20 2024-09-16T03:06:02.853 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 15 client.4508.objecter _send_op 20 to 6.1 on osd.1 2024-09-16T03:06:02.853 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 --> v1:172.21.15.45:6808/3228817795 -- osd_op(unknown.0.0:20 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x55ba50795560 con 0x55ba50633850 2024-09-16T03:06:02.853 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.853 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.839+0000 7f7b93692880 5 client.4508.objecter 1 in flight 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 <== osd.1 v1:172.21.15.45:6808/3228817795 20 ==== osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) ==== 149+0+0 (unknown 3106259312 0 0) 0x7f7b8c003bb0 con 0x55ba50633850 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 10 client.4508.objecter ms_dispatch 0x55ba505bdf00 osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 10 client.4508.objecter in handle_osd_op_reply 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 7 client.4508.objecter handle_osd_op_reply 20 ondisk uv 7 in 6.1 attempt 0 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 10 client.4508.objecter op 0 rval 0 len 0 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 15 client.4508.objecter handle_osd_op_reply completed tid 20 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 15 client.4508.objecter _finish_op 20 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 4 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 15 client.4508.objecter _session_op_remove 1 20 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 5 client.4508.objecter 0 in flight 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (3 ms) 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 20 librados: complete_aio_write 0x55ba50638de0 2024-09-16T03:06:02.854 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 10 librados: watch_flush enter 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 10 librados: watch_flush exit 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 10 client.4508.objecter close_session for osd.1 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 >> v1:172.21.15.45:6808/3228817795 conn(0x55ba50633850 legacy=0x55ba50635c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 20 client.4508.objecter put_session s=0x55ba506e2c90 osd=1 1 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 20 client.4508.objecter shutdown clearing up homeless session... 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 10 client.4508.objecter successfully canceled tick 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 >> v1:172.21.15.46:6812/149360280 conn(0x7f7b7403c1b0 legacy=0x7f7b7403e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 10 monclient: shutdown 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 >> v1:172.21.15.45:6790/0 conn(0x55ba50640300 legacy=0x55ba506e19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b9199d640 1 -- 172.21.15.46:0/1554543546 reap_dead start 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 shutdown_connections 2024-09-16T03:06:02.855 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 >> 172.21.15.46:0/1554543546 conn(0x55ba5063f810 msgr2=0x55ba506f4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:06:02.856 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 shutdown_connections 2024-09-16T03:06:02.856 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 1 -- 172.21.15.46:0/1554543546 wait complete. 2024-09-16T03:06:02.856 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.843+0000 7f7b93692880 1 librados: shutdown 2024-09-16T03:06:02.856 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [----------] 12 tests from AsioRados (116 ms total) 2024-09-16T03:06:02.856 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: 2024-09-16T03:06:02.856 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [----------] Global test environment tear-down 2024-09-16T03:06:02.856 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1340 ms total) 2024-09-16T03:06:02.856 INFO:tasks.workunit.client.0.smithi046.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-16T03:06:03.206 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-16T03:06:03.206 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [----------] Global test environment set-up. 2024-09-16T03:06:03.206 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-16T03:06:03.206 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-16T03:06:03.206 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosList.ListObjects (188 ms) 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (24 ms) 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo1 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo2 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo3 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo4 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo5 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo1 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: myset foo6,foo7 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo7 2024-09-16T03:06:03.207 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo6 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns1:foo4 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns1:foo5 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns2:foo7 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns2:foo6 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns1:foo1 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: :foo1 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: :foo2 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: :foo3 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (71 ms) 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 1 0 2024-09-16T03:06:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 10 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 13 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 7 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 14 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 0 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 15 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 11 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 5 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 8 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 6 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 3 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 4 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 12 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 9 0 2024-09-16T03:06:03.209 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 2 0 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (92 ms) 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: x cursor=MIN 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=1 cursor=9:02547ec2:::1:head 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=10 cursor=9:52ea6a34:::10:head 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=13 cursor=9:566253c9:::13:head 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=7 cursor=9:5c6b0b28:::7:head 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=14 cursor=9:62a1935d:::14:head 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=0 cursor=9:6cac518f:::0:head 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=15 cursor=9:863748b0:::15:head 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=11 cursor=9:89d3ae78:::11:head 2024-09-16T03:06:03.210 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=5 cursor=9:b29083e3:::5:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=8 cursor=9:bd63b0f1:::8:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=6 cursor=9:c4fdafeb:::6:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=3 cursor=9:cfc208b3:::3:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=4 cursor=9:d83876eb:::4:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=12 cursor=9:de5d7c5f:::12:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=9 cursor=9:e960b815:::9:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > oid=2 cursor=9:f905c69b:::2:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=1 cursor=9:02547ec2:::1:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=10 cursor=9:52ea6a34:::10:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-09-16T03:06:03.211 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-09-16T03:06:03.212 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=13 cursor=9:566253c9:::13:head 2024-09-16T03:06:03.212 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:566253c9:::13:head 2024-09-16T03:06:03.212 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-09-16T03:06:03.212 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=7 cursor=9:5c6b0b28:::7:head 2024-09-16T03:06:03.212 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-09-16T03:06:03.212 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-09-16T03:06:03.212 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=14 cursor=9:62a1935d:::14:head 2024-09-16T03:06:03.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-09-16T03:06:03.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-09-16T03:06:03.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=0 cursor=9:6cac518f:::0:head 2024-09-16T03:06:03.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-09-16T03:06:03.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-09-16T03:06:03.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=15 cursor=9:863748b0:::15:head 2024-09-16T03:06:03.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:863748b0:::15:head 2024-09-16T03:06:03.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=11 cursor=9:89d3ae78:::11:head 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=5 cursor=9:b29083e3:::5:head 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=8 cursor=9:bd63b0f1:::8:head 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=6 cursor=9:c4fdafeb:::6:head 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=3 cursor=9:cfc208b3:::3:head 2024-09-16T03:06:03.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=4 cursor=9:d83876eb:::4:head 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=12 cursor=9:de5d7c5f:::12:head 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=9 cursor=9:e960b815:::9:head 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:e960b815:::9:head 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : oid=2 cursor=9:f905c69b:::2:head 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-09-16T03:06:03.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:f905c69b:::2:head expected=9:f905c69b:::2:head 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=2 expected=2 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:e960b815:::9:head 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:e960b815:::9:head expected=9:e960b815:::9:head 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=9 expected=9 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:de5d7c5f:::12:head expected=9:de5d7c5f:::12:head 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=12 expected=12 2024-09-16T03:06:03.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:d83876eb:::4:head expected=9:d83876eb:::4:head 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=4 expected=4 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:cfc208b3:::3:head expected=9:cfc208b3:::3:head 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=3 expected=3 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:c4fdafeb:::6:head expected=9:c4fdafeb:::6:head 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=6 expected=6 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-09-16T03:06:03.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:bd63b0f1:::8:head expected=9:bd63b0f1:::8:head 2024-09-16T03:06:03.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-09-16T03:06:03.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=8 expected=8 2024-09-16T03:06:03.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-09-16T03:06:03.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:b29083e3:::5:head expected=9:b29083e3:::5:head 2024-09-16T03:06:03.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-09-16T03:06:03.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=5 expected=5 2024-09-16T03:06:03.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : curso handler_error: Running main() from gmock_main.cc 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: [----------] Global test environment set-up. 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2839 ms) 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: [----------] 1 test from neocls_handler_error (2839 ms total) 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: [----------] Global test environment tear-down 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2840 ms total) 2024-09-16T03:06:04.644 INFO:tasks.workunit.client.0.smithi046.stdout: handler_error: [ PASSED ] 1 test. 2024-09-16T03:06:04.649 INFO:tasks.workunit.client.0.smithi046.stdout: cls: Running main() from gmock_main.cc 2024-09-16T03:06:04.649 INFO:tasks.workunit.client.0.smithi046.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-09-16T03:06:04.649 INFO:tasks.workunit.client.0.smithi046.stdout: cls: [----------] Global test environment set-up. 2024-09-16T03:06:04.649 INFO:tasks.workunit.client.0.smithi046.stdout: cls: [----------] 1 test from NeoRadosCls 2024-09-16T03:06:04.649 INFO:tasks.workunit.client.0.smithi046.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-16T03:06:04.649 INFO:tasks.workunit.client.0.smithi046.stdout: cls: [ OK ] NeoRadosCls.DNE (2868 ms) 2024-09-16T03:06:04.649 INFO:tasks.workunit.client.0.smithi046.stdout: cls: [----------] 1 test from NeoRadosCls (2868 ms total) 2024-09-16T03:06:04.649 INFO:tasks.workunit.client.0.smithi046.stdout: cls: 2024-09-16T03:06:04.649 INFO:tasks.workunit.client.0.smithi046.stdout: cls: [----------] Global test environment tear-down 2024-09-16T03:06:04.650 INFO:tasks.workunit.client.0.smithi046.stdout: cls: [==========] 1 test from 1 test suite ran. (2868 ms total) 2024-09-16T03:06:04.650 INFO:tasks.workunit.client.0.smithi046.stdout: cls: [ PASSED ] 1 test. 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (50 ms) 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (60 ms) 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2866 ms) 2024-09-16T03:06:04.658 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2976 ms total) 2024-09-16T03:06:04.659 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: 2024-09-16T03:06:04.659 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-16T03:06:04.659 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2976 ms total) 2024-09-16T03:06:04.659 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-16T03:06:04.661 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-16T03:06:04.661 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-16T03:06:04.661 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (307 ms) 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (5 ms) 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (33 ms) 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (26 ms) 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-16T03:06:04.662 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (7 ms) 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (6 ms) 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5 ms) 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (8 ms) 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (4 ms) 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6 ms) 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (5 ms) 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-16T03:06:04.663 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (12 ms) 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (10 ms) 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (9 ms) 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (452 ms total) 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2962 ms total) 2024-09-16T03:06:04.664 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-16T03:06:04.812 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (65 ms) 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (42 ms) 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2881 ms) 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.676695+0000 mon.b [WRN] Health check failed: 27 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T03:06:04.813 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.748725+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/4088086294' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi046-28514-16"} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.754309+0000 mon.b [INF] from='client.4718 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi046-28514-16"} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.756624+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/4088086294' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi046-28514-16"} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.759648+0000 mon.b [INF] from='client.4718 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi046-28514-16"} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.766486+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/4088086294' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi046-28514-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.776173+0000 mon.b [INF] from='client.4718 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi046-28514-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.784400+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/2668668031' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi046-28522-24"} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.793478+0000 mon.b [INF] from='client.4733 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi046-28522-24"} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.799012+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/2668668031' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi046-28522-24"} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.799378+0000 mon.b [INF] from='client.4733 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi046-28522-24"} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.805800+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/2668668031' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi046-28522-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:03.808557+0000 mon.b [INF] from='client.4733 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi046-28522-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:04.814 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.630910+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/2776566801' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi046-28898-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi046-28898-1"}]': finished 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.631029+0000 mon.b [INF] from='client.4667 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi046-29339-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi046-29339-1"}]': finished 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.631118+0000 mon.b [INF] from='client.4679 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi046-29396-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi046-29396-1"}]': finished 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.631209+0000 mon.b [INF] from='client.4520 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi046-28536-1","var":"pgp_num","val":"11"}]': finished 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.631299+0000 mon.b [INF] from='client.4718 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi046-28514-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.631403+0000 mon.b [INF] from='client.4733 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi046-28522-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.636542+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/4088086294' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi046-28514-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi046-28514-16"} : dispatch 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.636804+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/2668668031' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi046-28522-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi046-28522-24"} : dispatch 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.645517+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/3985648601' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-28678-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.678015+0000 mon.b [INF] from='client.4718 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi046-28514-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi046-28514-16"} : dispatch 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.678587+0000 mon.b [INF] from='client.4733 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi046-28522-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi046-28522-24"} : dispatch 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.679301+0000 mon.b [INF] from='client.4742 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-28678-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:04.815 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.680996+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/4046841691' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi046-28506-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:04.970 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.682182+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/4089445787' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWrite_smithi046-28498-2","app": list_parallel: process_1_[29079]: starting. 2024-09-16T03:06:04.970 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_1_[29079]: creating pool ceph_test_rados_list_parallel.smithi046-29062 2024-09-16T03:06:04.970 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_1_[29079]: created object 0... 2024-09-16T03:06:04.970 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_1_[29079]: created object 25... 2024-09-16T03:06:04.970 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_1_[29079]: created object 49... 2024-09-16T03:06:04.970 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_1_[29079]: finishing. 2024-09-16T03:06:04.970 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_1_[29079]: shutting down. 2024-09-16T03:06:04.970 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_2_[29080]: starting. 2024-09-16T03:06:04.970 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_2_[29080]: listing objects. 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_2_[29080]: listed object 0... 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_2_[29080]: listed object 25... 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_2_[29080]: saw 50 objects 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_2_[29080]: shutting down. 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_3_[29798]: starting. 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_3_[29798]: creating pool ceph_test_rados_list_parallel.smithi046-29062 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_3_[29798]: created object 0... 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_3_[29798]: created object 25... 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_3_[29798]: created object 49... 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_3_[29798]: finishing. 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_3_[29798]: shutting down. 2024-09-16T03:06:04.971 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_4_[29799]: starting. 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_4_[29799]: listing objects. 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_4_[29799]: listed object 0... 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_4_[29799]: listed object 25... 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_4_[29799]: saw 43 objects 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_4_[29799]: shutting down. 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_5_[29800]: starting. 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_5_[29800]: removed 25 objects... 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_5_[29800]: removed half of the objects 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_5_[29800]: removed 50 objects... 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_5_[29800]: removed 50 objects 2024-09-16T03:06:04.972 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_5_[29800]: shutting down. 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_6_[29851]: starting. 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_6_[29851]: creating pool ceph_test_rados_list_parallel.smithi046-29062 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_6_[29851]: created object 0... 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_6_[29851]: created object 25... 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_6_[29851]: created object 49... 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_6_[29851]: finishing. 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_6_[29851]: shutting down. 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_7_[29852]: starting. 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_7_[29852]: listing objects. 2024-09-16T03:06:04.973 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_7_[29852]: listed object 0... 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_7_[29852]: listed object 25... 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_7_[29852]: listed object 50... 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_7_[29852]: saw 56 objects 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_7_[29852]: shutting down. 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_8_[29853]: starting. 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_8_[29853]: added 25 objects... 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_8_[29853]: added half of the objects 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_8_[29853]: added 50 objects... 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_8_[29853]: added 50 objects 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_8_[29853]: shutting down. 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:04.974 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.166 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.166 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_9_[29902]: starting. 2024-09-16T03:06:05.166 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_9_[29902]: creating pool ceph_test_rados_list_parallel.smithi046-29062 2024-09-16T03:06:05.166 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_9_[29902]: created object 0... 2024-09-16T03:06:05.166 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_9_[29902]: created object 25... 2024-09-16T03:06:05.166 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_9_[29902]: created object 49... 2024-09-16T03:06:05.166 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_9_[29902]: finishing. 2024-09-16T03:06:05.166 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_9_[29902]: shutting down. 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_10_[29903]: starting. 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_10_[29903]: listing objects. 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_10_[29903]: listed object 0... 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_10_[29903]: listed object 25... 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_10_[29903]: listed object 50... 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_10_[29903]: listed object 75... 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_10_[29903]: saw 100 objects 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_10_[29903]: shutting down. 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.167 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_13_[29906]: starting. 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_13_[29906]: removed 25 objects... 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_13_[29906]: removed half of the objects 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_13_[29906]: removed 50 objects... 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_13_[29906]: removed 50 objects 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_13_[29906]: shutting down. 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_12_[29905]: starting. 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_12_[29905]: added 25 objects... 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_12_[29905]: added half of the objects 2024-09-16T03:06:05.168 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_12_[29905]: added 50 objects... 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_12_[29905]: added 50 objects 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_12_[29905]: shutting down. 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_11_[29904]: starting. 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_11_[29904]: added 25 objects... 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_11_[29904]: added half of the objects 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_11_[29904]: added 50 objects... 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_11_[29904]: added 50 objects 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_11_[29904]: shutting down. 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.169 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_14_[30235]: starting. 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_14_[30235]: creating pool ceph_test_rados_list_parallel.smithi046-29062 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_14_[30235]: created object 0... 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_14_[30235]: created object 25... 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_14_[30235]: created object 49... 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_14_[30235]: finishing. 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_14_[30235]: shutting down. 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: starting. 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: listing objects. 2024-09-16T03:06:05.170 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: listed object 0... 2024-09-16T03:06:05.683 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: listed object 25... 2024-09-16T03:06:05.683 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: listed object 50... 2024-09-16T03:06:05.683 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: listed object 75... 2024-09-16T03:06:05.683 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: listed object 100... 2024-09-16T03:06:05.683 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: listed object 125... 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: saw 150 objects 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_15_[30236]: shutting down. 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_16_[30237]: starting. 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_16_[30237]: added 25 objects... 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_16_[30237]: added half of the objects 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_16_[30237]: added 50 objects... 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_16_[30237]: added 50 objects 2024-09-16T03:06:05.684 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: process_16_[30237]: shutting down. 2024-09-16T03:06:05.685 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.685 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.685 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.685 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.685 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******************************* 2024-09-16T03:06:05.685 INFO:tasks.workunit.client.0.smithi046.stdout: list_parallel: ******* SUCCESS ********** 2024-09-16T03:06:05.710 INFO:tasks.workunit.client.0.smithi046.stdout: "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.753307+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/1017022288' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi046-28787-7"} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.753742+0000 mon.b [INF] from='client.4829 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi046-28787-7"} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.759638+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/1017022288' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi046-28787-7"} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.761262+0000 mon.b [INF] from='client.4829 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi046-28787-7"} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.767328+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/1017022288' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi046-28787-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.775509+0000 mon.b [INF] from='client.4829 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi046-28787-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.808375+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/2768084377' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi046-28758-7"} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.808926+0000 client.admin [INF] onexx 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.809395+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/3540535681' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.819139+0000 mon.b [INF] from='client.4853 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi046-28758-7"} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.819345+0000 mon.b [INF] from='client.4877 ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.822951+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/2768084377' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi046-28758-7"} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.828012+0000 mon.b [INF] from='client.4853 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi046-28758-7"} : dispatch 2024-09-16T03:06:05.711 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.829257+0000 mon.b [INF] from='client.4853 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi046-28758-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:05.712 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:04.829306+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/2768084377' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi046-28758-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:05.712 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.641819+0000 mon.b [INF] from='client.4742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-28678-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:06:07.706 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.642042+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/4046841691' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi046-28506-2","app": "rados","yes_i_rea delete_pools_parallel: process_1_[29152]: starting. 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_1_[29152]: creating pool ceph_test_rados_delete_pools_parallel.smithi046-29090 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_1_[29152]: created object 0... 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_1_[29152]: created object 25... 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_1_[29152]: created object 49... 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_1_[29152]: finishing. 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_1_[29152]: shutting down. 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_2_[29153]: starting. 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_2_[29153]: deleting pool ceph_test_rados_delete_pools_parallel.smithi046-29090 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_2_[29153]: shutting down. 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_3_[29917]: starting. 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_3_[29917]: creating pool ceph_test_rados_delete_pools_parallel.smithi046-29090 2024-09-16T03:06:07.707 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_3_[29917]: created object 0... 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_3_[29917]: created object 25... 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_3_[29917]: created object 49... 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_3_[29917]: finishing. 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_3_[29917]: shutting down. 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_5_[29919]: starting. 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_5_[29919]: listing objects. 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_5_[29919]: listed object 0... 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_5_[29919]: listed object 25... 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_5_[29919]: saw 50 objects 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_5_[29919]: shutting down. 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:06:07.708 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_4_[29918]: starting. 2024-09-16T03:06:07.709 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_4_[29918]: deleting pool ceph_test_rados_delete_pools_parallel.smithi046-29090 2024-09-16T03:06:07.709 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: process_4_[29918]: shutting down. 2024-09-16T03:06:07.709 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:06:07.709 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:06:07.709 INFO:tasks.workunit.client.0.smithi046.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-16T03:06:07.709 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_1_[29141]: starting. 2024-09-16T03:06:07.709 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_1_[29141]: creating pool ceph_test_rados_open_pools_parallel.smithi046-29075 2024-09-16T03:06:07.709 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_1_[29141]: created object 0... 2024-09-16T03:06:07.709 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_1_[29141]: created object 25... 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_1_[29141]: created object 49... 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_1_[29141]: finishing. 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_1_[29141]: shutting down. 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_2_[29142]: starting. 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_2_[29142]: rados_pool_create. 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_2_[29142]: rados_ioctx_create. 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_2_[29142]: shutting down. 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: ******************************* 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_3_[29924]: starting. 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_3_[29924]: creating pool ceph_test_rados_open_pools_parallel.smithi046-29075 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_3_[29924]: created object 0... 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_3_[29924]: created object 25... 2024-09-16T03:06:07.710 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_3_[29924]: created object 49... 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_3_[29924]: finishing. 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_3_[29924]: shutting down. 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: ******************************* 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_4_[29927]: starting. 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_4_[29927]: rados_pool_create. 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_4_[29927]: rados_ioctx_create. 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: process_4_[29927]: shutting down. 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: ******************************* 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: ******************************* 2024-09-16T03:06:07.711 INFO:tasks.workunit.client.0.smithi046.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-16T03:06:07.886 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-16T03:06:07.886 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-09-16T03:06:07.886 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-16T03:06:07.886 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-09-16T03:06:07.886 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: seed 28522 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (154 ms) 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (8 ms) 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (4 ms) 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (346 ms) 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-09-16T03:06:07.887 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (4 ms) 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (4 ms) 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (8 ms) 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (6 ms) 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-16T03:06:07.888 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (8 ms) 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (7 ms) 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (9 ms) 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (20 ms) 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (11 ms) 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (6 ms) 2024-09-16T03:06:07.889 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (8 ms) 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (5 ms) 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (8 ms) 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (634 ms total) 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1181 ms) 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (22 ms) 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-16T03:06:07.890 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3 ms) 2024-09-16T03:06:08.701 INFO:tasks.workunit.client.0.smithi046.stdout: a cmd: Running main() from gmock_main.cc 2024-09-16T03:06:08.701 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:06:08.701 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [----------] Global test environment set-up. 2024-09-16T03:06:08.701 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-16T03:06:08.701 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-16T03:06:08.701 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1894 ms) 2024-09-16T03:06:08.702 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-16T03:06:08.702 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2011 ms) 2024-09-16T03:06:08.702 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-16T03:06:08.702 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3007 ms) 2024-09-16T03:06:08.702 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6912 ms total) 2024-09-16T03:06:08.702 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: 2024-09-16T03:06:08.702 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [----------] Global test environment tear-down 2024-09-16T03:06:08.702 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6912 ms total) 2024-09-16T03:06:08.702 INFO:tasks.workunit.client.0.smithi046.stdout: cmd: [ PASSED ] 3 tests. 2024-09-16T03:06:08.735 INFO:tasks.workunit.client.0.smithi046.stdout:lly_mean_it": true}]': finished 2024-09-16T03:06:08.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.642168+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/4089445787' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi046-28498-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.642267+0000 mon.b [INF] from='client.4829 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi046-28787-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.642427+0000 mon.b [INF] from='client.4853 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi046-28758-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.651597+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/1017022288' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi046-28787-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi046-28787-7"} : dispatch 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.652448+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/2768084377' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi046-28758-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi046-28758-7"} : dispatch 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.674324+0000 mon.b [INF] from='client.4829 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi046-28787-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi046-28787-7"} : dispatch 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.675751+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/4153997469' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi046-29339-1"} : dispatch 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.676201+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/3041762283' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-29001-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.677446+0000 mon.b [INF] from='client.4853 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi046-28758-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi046-28758-7"} : dispatch 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.687919+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/3456483931' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-28673-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.699541+0000 mon.b [INF] from='client.4667 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi046-29339-1"} : dispatch 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.699874+0000 mon.b [INF] from='client.4802 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-29001-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:08.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:05.700277+0000 mon.b [INF] from='client.4841 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-28673-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:06.709359+0000 mon.b [INF] from='client.4877 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.424636+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.0"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.426602+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.1"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.428269+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.2"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.429805+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.3"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.431507+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.4"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.433041+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.5"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.434692+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.6"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.436332+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.7"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.437973+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.8"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.439707+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/567650785' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "13.9"} : dispatch 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.660366+0000 mon.b [INF] from='client.4829 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi046-28787-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi046-28787-7"}]': finished 2024-09-16T03:06:08.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.660581+0000 mon.b [INF] from='client.4853 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi046-28758-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi046-28758-7"}]': finished 2024-09-16T03:06:08.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.660721+0000 mon.b [INF] from='client.4667 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi046-29339-1"}]': finished 2024-09-16T03:06:08.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.660810+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/2776566801' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi046-28898-1"}]': finished 2024-09-16T03:06:08.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.660898+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/4046841691' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi046-28506-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:06:08.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.660980+0000 mon.b [INF] from='client.4679 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi046-29396-1"}]': finished 2024-09-16T03:06:08.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.661066+0000 mon.b [INF] from='client.4742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-28678-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:06:08.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.670953+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/3295725004' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi046-29396-1"} : dispatch 2024-09-16T03:06:08.780 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.677213+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/3985648601' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi046-28678-4", "tierpool": "test-rados-api-smithi046-28678-4-cache"} : dispatch 2024-09-16T03:06:08.780 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.690016+0000 mon.b [INF] from='client.4679 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi046-29396-1"} : dispatch 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.691543+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/2776566801' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi046-28898-1"} : dispatch 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.692129+0000 mon.b [INF] from='client.4742 ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi046-28678-4", "tierpool": "test-rados-api-smithi046-28678-4-cache"} : dispatch 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.693194+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/3064130256' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi046-28498-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.737615+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/4046841691' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi046-28506-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.739941+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/149278514' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi046-29339-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.740254+0000 mon.b [INF] from='client.5063 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi046-29339-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.932462+0000 mon.b [INF] from='client.4733 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi046-28522-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:07.932750+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/2668668031' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi046-28522-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.669918+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/1119945314' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi046-28577-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi046-28577-10"}]': finished 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.670174+0000 mon.b [INF] from='client.4931 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi046-28543-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi046-28543-10"}]': finished 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.670406+0000 mon.b [INF] from='client.4679 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi046-29396-1"}]': finished 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.670583+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/2776566801' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi046-28898-1"}]': finished 2024-09-16T03:06:08.781 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.670740+0000 mon.b [INF] from='client.4742 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi046-28678-4", "tierpool": "test-rados-api-smithi046-28678-4-cache"}]': finished 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.670970+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/3064130256' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi046-28498-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.671147+0000 mon.b [INF] from='client.? v1:172.21.15.46:0/4046841691' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi046-28506-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.671314+0000 mon.b [INF] from='client.5063 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi046-29339-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.671528+0000 mon.b [INF] from='client.4733 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi046-28522-24", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.681857+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/3985648601' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi046-28678-4", "overlaypool": "test-rados-api-smithi046-28678-4-cache"} : dispatch 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.682266+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/149278514' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi046-29339-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi046-29339-2"} : dispatch 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.692801+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/643305843' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-29001-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.693444+0000 mon.b [INF] from='client.4742 ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi046-28678-4", "overlaypool": "test-rados-api-smithi046-28678-4-cache"} : dispatch 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.694005+0000 mon.b [INF] from='client.5063 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi046-29339-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi046-29339-2"} : dispatch 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.698629+0000 mon.b [INF] from='client.5090 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-29001-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.700140+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/3402756999' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-28673-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.701212+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/4088086294' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi046-28514-16"} : dispatch 2024-09-16T03:06:08.782 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.704377+0000 mon.b [INF] from='client.5087 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi046-28673-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T03:06:08.783 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.706547+0000 mon.b [INF] from='client.4718 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi046-28514-16"} : dispatch 2024-09-16T03:06:08.783 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.733477+0000 mon.c [INF] from='client.? v1:172.21.15.46:0/3540535681' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T03:06:10.714 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.738006+0000 client.ad api_io: Running main() from gmock_main.cc 2024-09-16T03:06:10.714 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-16T03:06:10.714 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [----------] Global test environment set-up. 2024-09-16T03:06:10.714 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-16T03:06:10.714 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-16T03:06:10.714 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (84 ms) 2024-09-16T03:06:10.714 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-16T03:06:10.714 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: no timeout :/ 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: no timeout :/ 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: no timeout :/ 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: no timeout :/ 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: no timeout :/ 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (88 ms) 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (5 ms) 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.Checksum (2 ms) 2024-09-16T03:06:10.715 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (3 ms) 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (3 ms) 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (4 ms) 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (1 ms) 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3 ms) 2024-09-16T03:06:10.716 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (3 ms) 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.RmXattr (9 ms) 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIo.XattrIter (5 ms) 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [----------] 14 tests from LibRadosIo (214 ms total) 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1184 ms) 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (15 ms) 2024-09-16T03:06:10.717 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (6 ms) 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (5 ms) 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (5 ms) 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (5 ms) 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (5 ms) 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-16T03:06:10.718 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (12 ms) 2024-09-16T03:06:10.719 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-16T03:06:10.719 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-09-16T03:06:10.719 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1249 ms total) 2024-09-16T03:06:10.719 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: 2024-09-16T03:06:10.719 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [----------] Global test environment tear-down 2024-09-16T03:06:10.719 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9235 ms total) 2024-09-16T03:06:10.719 INFO:tasks.workunit.client.0.smithi046.stdout: api_io: [ PASSED ] 24 tests. 2024-09-16T03:06:11.722 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: Running main() from gmock_main.cc 2024-09-16T03:06:11.722 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-16T03:06:11.722 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-16T03:06:11.722 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-16T03:06:11.722 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-16T03:06:11.722 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: handle_notify cookie 94167722638448 notify_id 98784247808 notifier_gid 4706 2024-09-16T03:06:11.722 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2778 ms) 2024-09-16T03:06:11.722 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: Trying... 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: handle_notify cookie 94167734669024 notify_id 111669149696 notifier_gid 4826 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: Waiting for 3.000000000s 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: Timed out. 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: Flushing... 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: Flushed... 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7066 ms) 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (9844 ms total) 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (9844 ms total) 2024-09-16T03:06:11.723 INFO:tasks.workunit.client.0.smithi046.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-16T03:06:11.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: Running main() from gmock_main.cc 2024-09-16T03:06:11.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-16T03:06:11.734 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [----------] Global test environment set-up. 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ OK ] LibRadosStat.Stat (289 ms) 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (37 ms) 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ OK ] LibRadosStat.StatNS (10 ms) 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (3 ms) 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [----------] 5 tests from LibRadosStat (340 ms total) 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: 2024-09-16T03:06:11.735 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1162 ms) 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (10 ms) 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1174 ms total) 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [----------] Global test environment tear-down 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (10154 ms total) 2024-09-16T03:06:11.736 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: seed 28787 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (286 ms) 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (37 ms) 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (22 ms) 2024-09-16T03:06:11.737 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (11 ms) 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (357 ms total) 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1166 ms) 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (3 ms) 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (8 ms) 2024-09-16T03:06:11.738 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1178 ms total) 2024-09-16T03:06:11.739 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: 2024-09-16T03:06:11.739 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-16T03:06:11.739 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10131 ms total) 2024-09-16T03:06:11.739 INFO:tasks.workunit.client.0.smithi046.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-16T03:06:11.739 INFO:tasks.workunit.client.0.smithi046.stdout: list: Running main() from gmock_main.cc 2024-09-16T03:06:11.739 INFO:tasks.workunit.client.0.smithi046.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:06:11.739 INFO:tasks.workunit.client.0.smithi046.stdout: list: [----------] Global test environment set-up. 2024-09-16T03:06:11.739 INFO:tasks.workunit.client.0.smithi046.stdout: list: [----------] 3 tests from NeoradosList 2024-09-16T03:06:11.739 INFO:tasks.workunit.client.0.smithi046.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: [ OK ] NeoradosList.ListObjects (2838 ms) 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3036 ms) 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: [ OK ] NeoradosList.ListObjectsMany (4026 ms) 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: [----------] 3 tests from NeoradosList (9901 ms total) 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: [----------] Global test environment tear-down 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: [==========] 3 tests from 1 test suite ran. (9901 ms total) 2024-09-16T03:06:11.740 INFO:tasks.workunit.client.0.smithi046.stdout: list: [ PASSED ] 3 tests. 2024-09-16T03:06:11.810 INFO:tasks.workunit.client.0.smithi046.stdout:min [INF] threexx 2024-09-16T03:06:11.810 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: got: 2024-09-16T03:06:08.738762+0000 mon.b [INF] from='client.4877 ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T03:06:11.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7158 ms) 2024-09-16T03:06:11.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (10146 ms total) 2024-09-16T03:06:11.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: 2024-09-16T03:06:11.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-16T03:06:11.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (10146 ms total) 2024-09-16T03:06:11.811 INFO:tasks.workunit.client.0.smithi046.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-16T03:06:14.262 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: Running main() from gmock_main.cc 2024-09-16T03:06:14.262 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-16T03:06:14.262 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [----------] Global test environment set-up. 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5856 ms) 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6581 ms) 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12438 ms total) 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [----------] Global test environment tear-down 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12439 ms total) 2024-09-16T03:06:14.263 INFO:tasks.workunit.client.0.smithi046.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: Running main() from gmock_main.cc 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [----------] Global test environment set-up. 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (65 ms) 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLock.LockShared (96 ms) 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-16T03:06:14.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1133 ms) 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1010 ms) 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5 ms) 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5 ms) 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (6 ms) 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [----------] 8 tests from LibRadosLock (2323 ms total) 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-16T03:06:14.270 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1189 ms) 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (4 ms) 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1044 ms) 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1020 ms) 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (10 ms) 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (7 ms) 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (9 ms) 2024-09-16T03:06:14.271 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-16T03:06:14.272 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (4 ms) 2024-09-16T03:06:14.272 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3287 ms total) 2024-09-16T03:06:14.272 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: 2024-09-16T03:06:14.272 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [----------] Global test environment tear-down 2024-09-16T03:06:14.272 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (12764 ms total) 2024-09-16T03:06:14.272 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-16T03:06:14.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-16T03:06:14.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T03:06:14.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-16T03:06:14.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: seed 28577 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (140 ms) 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (12 ms) 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1167 ms) 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1005 ms) 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (4 ms) 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-09-16T03:06:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (6 ms) 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2342 ms total) 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1100 ms) 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (81 ms) 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1048 ms) 2024-09-16T03:06:14.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1016 ms) 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (6 ms) 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (8 ms) 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (10 ms) 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (6 ms) 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3275 ms total) 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (12757 ms total) 2024-09-16T03:06:14.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-16T03:06:14.427 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-16T03:06:14.428 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T03:06:14.428 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-16T03:06:14.428 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-16T03:06:14.428 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-16T03:06:14.428 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: notify 2024-09-16T03:06:14.428 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1152 ms) 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (6 ms) 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1158 ms total) 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: notify 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (123 ms) 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-16T03:06:14.429 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: notify 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3542 ms) 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (5 ms) 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (5 ms) 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 146028888068 notifier_gid 5047 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5 ms) 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 146028888069 notifier_gid 5047 2024-09-16T03:06:14.430 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (5 ms) 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 146028888070 notifier_gid 5047 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 146028888071 notifier_gid 5047 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (7 ms) 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 146028888072 notifier_gid 5047 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (5 ms) 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 146028888073 notifier_gid 5047 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (6 ms) 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 146028888074 notifier_gid 5047 2024-09-16T03:06:14.431 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (6 ms) 2024-09-16T03:06:14.432 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-16T03:06:14.432 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: trying... 2024-09-16T03:06:15.276 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 146028888075 notifier_g pool: Running main() from gmock_main.cc 2024-09-16T03:06:15.276 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-16T03:06:15.276 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [----------] Global test environment set-up. 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ OK ] NeoRadosPools.PoolList (1823 ms) 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2001 ms) 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2026 ms) 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4006 ms) 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1536 ms) 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2022 ms) 2024-09-16T03:06:15.277 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [----------] 6 tests from NeoRadosPools (13415 ms total) 2024-09-16T03:06:15.278 INFO:tasks.workunit.client.0.smithi046.stdout: pool: 2024-09-16T03:06:15.278 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [----------] Global test environment tear-down 2024-09-16T03:06:15.278 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [==========] 6 tests from 1 test suite ran. (13415 ms total) 2024-09-16T03:06:15.278 INFO:tasks.workunit.client.0.smithi046.stdout: pool: [ PASSED ] 6 tests. 2024-09-16T03:06:18.311 INFO:tasks.workunit.client.0.smithi046.stdout:pi_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-16T03:06:18.311 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (3 ms) 2024-09-16T03:06:18.311 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (4 ms) 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (4 ms) 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (3 ms) 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (5 ms) 2024-09-16T03:06:18.312 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (5 ms) 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (10 ms) 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (6337 ms) 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1142 ms) 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (5 ms) 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-16T03:06:18.313 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (3 ms) 2024-09-16T03:06:18.314 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-16T03:06:18.314 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-09-16T03:06:18.314 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (8744 ms total) 2024-09-16T03:06:18.314 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: 2024-09-16T03:06:18.314 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-16T03:06:18.314 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (16822 ms total) 2024-09-16T03:06:18.314 INFO:tasks.workunit.client.0.smithi046.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-09-16T03:06:22.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-16T03:06:22.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-16T03:06:22.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-16T03:06:22.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-16T03:06:22.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (341 ms) 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94113033066112 err -107 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1011 ms) 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94113033068448 err -107 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1009 ms) 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_cb from 4568 notify_id 107374182401 cookie 94113033087344 2024-09-16T03:06:22.336 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (12 ms) 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_cb from 4568 notify_id 107374182400 cookie 94113033234480 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (13 ms) 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_cb from 4568 notify_id 107374182402 cookie 94113033234480 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (11 ms) 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_cb from 4568 notify_id 107374182400 cookie 94113033235408 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_cb from 4568 notify_id 107374182400 cookie 94113033275392 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (18 ms) 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-16T03:06:22.337 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_cb from 4568 notify_id 107374182401 cookie 94113033275392 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_cb from 4568 notify_id 111669149698 cookie 94113033275392 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3012 ms) 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94113033275392 err -107 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_cb from 4568 notify_id 141733920770 cookie 94113033275392 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5021 ms) 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94113033275392 err -107 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1014 ms) 2024-09-16T03:06:22.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11462 ms total) 2024-09-16T03:06:22.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: 2024-09-16T03:06:22.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-16T03:06:22.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-16T03:06:22.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T03:06:22.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1143 ms) 2024-09-16T03:06:22.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1143 ms total) 2024-09-16T03:06:22.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: 2024-09-16T03:06:22.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-16T03:06:22.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (20704 ms total) 2024-09-16T03:06:22.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-16T03:06:26.273 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-16T03:06:26.273 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2946 ms) 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3052 ms) 2024-09-16T03:06:26.274 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3025 ms) 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2535 ms) 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3024 ms) 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3045 ms) 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (6943 ms) 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (24570 ms total) 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (24570 ms total) 2024-09-16T03:06:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: Running main() from gmock_main.cc 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [----------] Global test environment set-up. 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (2767 ms) 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3048 ms) 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-16T03:06:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3028 ms) 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2536 ms) 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3011 ms) 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3057 ms) 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (6959 ms) 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (24406 ms total) 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [----------] Global test environment tear-down 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (24406 ms total) 2024-09-16T03:06:26.294 INFO:tasks.workunit.client.0.smithi046.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-16T03:06:27.267 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: Running main() from gmock_main.cc 2024-09-16T03:06:27.267 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:06:27.267 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [----------] Global test environment set-up. 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6856 ms) 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6558 ms) 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (12017 ms) 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [----------] 3 tests from NeoradosECList (25431 ms total) 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [----------] Global test environment tear-down 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (25431 ms total) 2024-09-16T03:06:27.268 INFO:tasks.workunit.client.0.smithi046.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-16T03:06:29.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [----------] Global test environment set-up. 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5067 ms) 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [ OK ] LibRadosService.RegisterLate (103 ms) 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: cluster: 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: id: 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 12 pool(s) do not have an application enabled 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: services: 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: mon: 3 daemons, quorum b,c (age 40s), out of quorum: a 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: mgr: x(active, since 83s) 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: osd: 6 osds: 6 up (since 78s), 6 in (since 87s) 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: data: 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: pools: 22 pools, 577 pgs 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: objects: 37 objects, 579 KiB 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: usage: 185 MiB used, 540 GiB / 540 GiB avail 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: pgs: 37.435% pgs unknown 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 25.650% pgs not active 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 216 unknown 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 213 active+clean 2024-09-16T03:06:29.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 148 creating+peering 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: io: 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: client: 2.0 KiB/s rd, 24 KiB/s wr, 3 op/s rd, 40 op/s wr 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: cluster: 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: id: 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 15 pool(s) do not have an application enabled 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: services: 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: mon: 3 daemons, quorum b,c (age 42s), out of quorum: a 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: mgr: x(active, since 85s) 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: osd: 6 osds: 6 up (since 80s), 6 in (since 89s) 2024-09-16T03:06:29.287 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: data: 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: pools: 27 pools, 761 pgs 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: objects: 19 objects, 578 KiB 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: usage: 185 MiB used, 540 GiB / 540 GiB avail 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: pgs: 69.382% pgs unknown 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 14.323% pgs not active 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 528 unknown 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 124 active+clean 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 109 creating+peering 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.288 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: io: 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: client: 1.7 KiB/s rd, 15 KiB/s wr, 1 op/s rd, 26 op/s wr 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2351 ms) 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [ OK ] LibRadosService.Status (20047 ms) 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [----------] 4 tests from LibRadosService (27568 ms total) 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [----------] Global test environment tear-down 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27568 ms total) 2024-09-16T03:06:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: api_service: [ PASSED ] 4 tests. 2024-09-16T03:06:30.332 INFO:tasks.workunit.client.0.smithi046.stdout:id 5047 2024-09-16T03:06:30.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: timed out 2024-09-16T03:06:30.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: flushing 2024-09-16T03:06:30.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: flushed 2024-09-16T03:06:30.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3005 ms) 2024-09-16T03:06:30.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-16T03:06:30.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: trying... 2024-09-16T03:06:30.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 158913789964 notifier_gid 5047 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: timed out 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: flushing 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: flushed 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3009 ms) 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: List watches 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: notify2 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155861312 notify_id 171798691853 notifier_gid 5047 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: notify2 done 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: watch_check 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: unwatch2 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: flushing 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: done 2024-09-16T03:06:30.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3010 ms) 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: List watches 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: notify2 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: handle_notify cookie 94490155832320 notify_id 184683593730 notifier_gid 5047 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: notify2 done 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: watch_check 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: unwatch2 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: flushing 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: done 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3030 ms) 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (15764 ms total) 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: 2024-09-16T03:06:30.334 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-16T03:06:30.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28684 ms total) 2024-09-16T03:06:30.335 INFO:tasks.workunit.client.0.smithi046.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-16T03:06:34.263 INFO:tasks.workunit.client.0.smithi046.stdout: misc: Running main() from gmock_main.cc 2024-09-16T03:06:34.263 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-16T03:06:34.263 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [----------] Global test environment set-up. 2024-09-16T03:06:34.263 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-16T03:06:34.263 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-16T03:06:34.263 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.Version (1824 ms) 2024-09-16T03:06:34.263 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2001 ms) 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.LongName (3032 ms) 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2997 ms) 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2546 ms) 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3040 ms) 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.Exec (3025 ms) 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-16T03:06:34.264 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2929 ms) 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3048 ms) 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2989 ms) 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2036 ms) 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2937 ms) 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [----------] 12 tests from NeoRadosMisc (32405 ms total) 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [----------] Global test environment tear-down 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [==========] 12 tests from 1 test suite ran. (32405 ms total) 2024-09-16T03:06:34.265 INFO:tasks.workunit.client.0.smithi046.stdout: misc: [ PASSED ] 12 tests. 2024-09-16T03:06:37.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-16T03:06:37.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5044 ms) 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (109 ms) 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20036 ms) 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: attempt 0 of 20 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: failed to deregister: 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: { 2024-09-16T03:06:37.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "epoch": 9, 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "modified": "2024-09-16T03:06:29.493269+0000", 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "services": { 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "laundry": { 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "daemons": { 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "summary": "", 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "close-test-pid28973": { 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "start_epoch": 8, 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "start_stamp": "2024-09-16T03:06:27.072441+0000", 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "gid": 12008, 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "addr": "172.21.15.46:0/3674780204", 2024-09-16T03:06:37.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "metadata": { 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "arch": "x86_64", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "ceph_release": "squid", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "distro": "ubuntu", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "distro_version": "22.04", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "foo": "bar", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "hostname": "smithi046", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "kernel_version": "5.15.0-121-generic", 2024-09-16T03:06:37.255 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "mem_swap_kb": "2097148", 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "mem_total_kb": "32753020", 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "os": "Linux", 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "this": "that" 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: }, 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: "task_status": {} 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: } 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: } 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: } 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: } 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: } 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: attempt 1 of 20 2024-09-16T03:06:37.256 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10353 ms) 2024-09-16T03:06:37.257 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (35542 ms total) 2024-09-16T03:06:37.257 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: 2024-09-16T03:06:37.257 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-16T03:06:37.257 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (35542 ms total) 2024-09-16T03:06:37.257 INFO:tasks.workunit.client.0.smithi046.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-16T03:06:39.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: Running main() from gmock_main.cc 2024-09-16T03:06:39.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-16T03:06:39.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [----------] Global test environment set-up. 2024-09-16T03:06:39.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-16T03:06:39.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-16T03:06:39.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ OK ] LibRadosPools.PoolList (3135 ms) 2024-09-16T03:06:39.284 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3011 ms) 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3019 ms) 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2551 ms) 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3041 ms) 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5000 ms) 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4958 ms) 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5050 ms) 2024-09-16T03:06:39.285 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-16T03:06:39.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7966 ms) 2024-09-16T03:06:39.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [----------] 9 tests from LibRadosPools (37731 ms total) 2024-09-16T03:06:39.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: 2024-09-16T03:06:39.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [----------] Global test environment tear-down 2024-09-16T03:06:39.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37732 ms total) 2024-09-16T03:06:39.286 INFO:tasks.workunit.client.0.smithi046.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-16T03:06:42.343 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-16T03:06:42.343 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] Global test environment set-up. 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: 2024-09-16T03:06:01.603+0000 7fdbcf67b880 -1 failed for service _ceph-mon._tcp 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: 2024-09-16T03:06:01.603+0000 7fdbcf67b880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (58 ms) 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-16T03:06:42.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5023 ms) 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5081 ms total) 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: started 0x55be18e30570 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: started 0x55be184ccf00 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: started 2 aios 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: waiting 0x55be18e30570 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: waiting 0x55be184ccf00 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: done. 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (6099 ms) 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (6099 ms total) 2024-09-16T03:06:42.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMisc.Exec (122 ms) 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (9 ms) 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2 ms) 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4919 ms) 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-16T03:06:42.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-16T03:06:42.347 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-16T03:06:42.347 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (22848 ms) 2024-09-16T03:06:42.347 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] 8 tests from LibRadosMisc (27901 ms total) 2024-09-16T03:06:42.347 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: 2024-09-16T03:06:42.347 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [----------] Global test environment tear-down 2024-09-16T03:06:42.347 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (40768 ms total) 2024-09-16T03:06:42.347 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: Running main() from gmock_main.cc 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: [----------] Global test environment set-up. 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.Limits (2836 ms) 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3054 ms) 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-16T03:06:43.274 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.ReadOp (3010 ms) 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.SparseRead (2541 ms) 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3028 ms) 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3044 ms) 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3020 ms) 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2921 ms) 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2988 ms) 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-16T03:06:43.275 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.Trunc (3051 ms) 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.Remove (2956 ms) 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3093 ms) 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.RmXattr (2906 ms) 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3003 ms) 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [----------] 14 tests from NeoRadosIo (41451 ms total) 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [----------] Global test environment tear-down 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [==========] 14 tests from 1 test suite ran. (41451 ms total) 2024-09-16T03:06:43.276 INFO:tasks.workunit.client.0.smithi046.stdout: io: [ PASSED ] 14 tests. 2024-09-16T03:06:43.278 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: Running main() from gmock_main.cc 2024-09-16T03:06:43.278 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-16T03:06:43.278 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [----------] Global test environment set-up. 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2785 ms) 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3039 ms) 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3024 ms) 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2532 ms) 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3026 ms) 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-16T03:06:43.279 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3062 ms) 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3007 ms) 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2912 ms) 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (2999 ms) 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3060 ms) 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2971 ms) 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3079 ms) 2024-09-16T03:06:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-16T03:06:43.281 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2900 ms) 2024-09-16T03:06:43.281 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-16T03:06:43.281 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3006 ms) 2024-09-16T03:06:43.281 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41403 ms total) 2024-09-16T03:06:43.281 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: 2024-09-16T03:06:43.281 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [----------] Global test environment tear-down 2024-09-16T03:06:43.281 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41403 ms total) 2024-09-16T03:06:43.281 INFO:tasks.workunit.client.0.smithi046.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-16T03:06:56.929 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T03:06:56.930 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:06:56.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:06:56.933 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:06:56.933 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:06:56.934 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:06:56.934 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:07:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-16T03:07:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-16T03:07:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-16T03:07:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-16T03:07:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-16T03:07:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2000 ms) 2024-09-16T03:07:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-16T03:07:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2010 ms) 2024-09-16T03:07:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2013 ms) 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2003 ms) 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8027 ms total) 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4162 ms) 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4105 ms) 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4850 ms) 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13117 ms total) 2024-09-16T03:07:03.201 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2906 ms) 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2070 ms) 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1911 ms) 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2040 ms) 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8927 ms total) 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-16T03:07:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4045 ms) 2024-09-16T03:07:03.203 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-16T03:07:03.203 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4002 ms) 2024-09-16T03:07:03.203 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8047 ms total) 2024-09-16T03:07:03.203 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: 2024-09-16T03:07:03.203 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-16T03:07:03.203 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (61620 ms total) 2024-09-16T03:07:03.203 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-16T03:07:11.934 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:07:11.935 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:07:11.935 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:07:11.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:07:12.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient: get_monmap_and_config 2024-09-16T03:07:12.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient: build_initial_monmap 2024-09-16T03:07:12.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient: monmap: 2024-09-16T03:07:12.067 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:12.067 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.067 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:12.065594+0000 2024-09-16T03:07:12.067 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:12.065594+0000 2024-09-16T03:07:12.067 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:12.068 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.068 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:12.068 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:12.068 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:12.068 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 1 Processor -- start 2024-09-16T03:07:12.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 1 -- start start 2024-09-16T03:07:12.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient: init 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient(hunting): picked mon.noname-c con 0x7f4694137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient(hunting): picked mon.noname-b con 0x7f4694138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient(hunting): picked mon.noname-a con 0x7f469413e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46940a4f90 con 0x7f469413e2a0 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46940a4cb0 con 0x7f4694137d20 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f469409dc70 con 0x7f4694138100 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient(hunting): authenticate will time out at 1122.714944s 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f46937fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f469413e2a0 0x7f4694140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33602/0 (socket says 172.21.15.45:33602) 2024-09-16T03:07:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f46937fe640 1 -- 172.21.15.45:0/2141792733 learned_addr learned my addr 172.21.15.45:0/2141792733 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 -- 172.21.15.45:0/2141792733 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2343305966 0 0) 0x7f46940a4f90 con 0x7f469413e2a0 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient(hunting): my global_id is 94140 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 -- 172.21.15.45:0/2141792733 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4680003530 con 0x7f469413e2a0 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 -- 172.21.15.45:0/2141792733 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2367494884 0 0) 0x7f4680003530 con 0x7f469413e2a0 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 -- 172.21.15.45:0/2141792733 >> v1:172.21.15.45:6790/0 conn(0x7f4694138100 legacy=0x7f469413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 -- 172.21.15.45:0/2141792733 >> v1:172.21.15.46:6789/0 conn(0x7f4694137d20 legacy=0x7f46940a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 monclient: found mon.noname-a 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 -- 172.21.15.45:0/2141792733 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469409cc50 con 0x7f469413e2a0 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: _finish_auth 0 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 -- 172.21.15.45:0/2141792733 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f46840016a0 con 0x7f469413e2a0 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 5 monclient: authenticate success, global_id 94140 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: dump: 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 -- 172.21.15.45:0/2141792733 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46840018c0 con 0x7f469413e2a0 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 1 -- 172.21.15.45:0/2141792733 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4684001d10 con 0x7f469413e2a0 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f469960b640 4 set_mon_vals no callback set 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.064+0000 7f4691ffb640 10 monclient: dump: 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient: shutdown 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 -- 172.21.15.45:0/2141792733 >> v1:172.21.15.45:6789/0 conn(0x7f469413e2a0 legacy=0x7f4694140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 -- 172.21.15.45:0/2141792733 shutdown_connections 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 -- 172.21.15.45:0/2141792733 >> 172.21.15.45:0/2141792733 conn(0x7f46940aa110 msgr2=0x7f46940aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 -- 172.21.15.45:0/2141792733 shutdown_connections 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 -- 172.21.15.45:0/2141792733 wait complete. 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient: build_initial_monmap 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient: monmap: 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 librados: starting msgr at 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 librados: starting objecter 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 Processor -- start 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 -- start start 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 librados: setting wanted keys 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 librados: calling monclient init 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient: init 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient(hunting): picked mon.noname-b con 0x7f4694137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient(hunting): picked mon.noname-a con 0x7f4694138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient(hunting): picked mon.noname-c con 0x7f469413e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46941cdb40 con 0x7f4694138100 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46940a1300 con 0x7f4694137d20 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f469413d960 con 0x7f469413e2a0 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f469960b640 10 monclient(hunting): authenticate will time out at 1122.718693s 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f46937fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f469413e2a0 0x7f46941d4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35530/0 (socket says 172.21.15.45:35530) 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f46927fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4694138100 0x7f46941cd430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33606/0 (socket says 172.21.15.45:33606) 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f46937fe640 1 -- 172.21.15.45:0/3778209596 learned_addr learned my addr 172.21.15.45:0/3778209596 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f468b7fe640 10 client.?.objecter ms_handle_connect 0x7f4694138100 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f468b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f468b7fe640 10 client.?.objecter ms_handle_connect 0x7f469413e2a0 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f468b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f468b7fe640 10 client.?.objecter ms_handle_connect 0x7f4694137d20 2024-09-16T03:07:12.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.068+0000 7f468b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 503248272 0 0) 0x7f46941cdb40 con 0x7f4694138100 2024-09-16T03:07:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): my global_id is 94143 2024-09-16T03:07:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4674004ec0 con 0x7f4694138100 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133198201 0 0) 0x7f469413d960 con 0x7f469413e2a0 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): my global_id is 28442 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46941cdb40 con 0x7f469413e2a0 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2903531284 0 0) 0x7f46941cdb40 con 0x7f469413e2a0 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f469413d960 con 0x7f469413e2a0 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4684001c10 con 0x7f469413e2a0 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): dump: 2024-09-16T03:07:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2886013252 0 0) 0x7f469413d960 con 0x7f469413e2a0 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 >> v1:172.21.15.46:6789/0 conn(0x7f4694137d20 legacy=0x7f46941ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 >> v1:172.21.15.45:6789/0 conn(0x7f4694138100 legacy=0x7f46941cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:07:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 monclient: found mon.c 2024-09-16T03:07:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469413c150 con 0x7f469413e2a0 2024-09-16T03:07:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:07:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:07:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f469960b640 5 monclient: authenticate success, global_id 28442 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f469960b640 10 monclient: _renew_subs 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f469960b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f469413b8b0 con 0x7f469413e2a0 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4684004c80 con 0x7f469413e2a0 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f469960b640 10 client.28442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4684004fd0 con 0x7f469413e2a0 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f469960b640 10 monclient: _renew_subs 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f469960b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f469413c8b0 con 0x7f469413e2a0 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f469960b640 1 librados: init done 2024-09-16T03:07:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f468b7fe640 10 monclient: dump: 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.076+0000 7f46917fa640 4 set_mon_vals no callback set 2024-09-16T03:07:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f469960b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4684011040 con 0x7f469413e2a0 2024-09-16T03:07:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f469960b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f469960b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4678005090 con 0x7f469413e2a0 2024-09-16T03:07:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(86..86 src has 1..86) ==== 7905+0+0 (unknown 2722104324 0 0) 0x7f468404c130 con 0x7f469413e2a0 2024-09-16T03:07:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f468b7fe640 10 client.28442.objecter ms_dispatch 0x7f4694000ba0 osd_map(86..86 src has 1..86) 2024-09-16T03:07:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f468b7fe640 3 client.28442.objecter handle_osd_map got epochs [86,86] > 0 2024-09-16T03:07:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f468b7fe640 3 client.28442.objecter handle_osd_map decoding full epoch 86 2024-09-16T03:07:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.080+0000 7f468b7fe640 20 client.28442.objecter dump_active .. 0 homeless 2024-09-16T03:07:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.084+0000 7f4692ffd640 10 monclient: get_auth_request con 0x7f467403e060 auth_method 0 2024-09-16T03:07:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.084+0000 7f468b7fe640 10 client.28442.objecter ms_handle_connect 0x7f467403e060 2024-09-16T03:07:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.088+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4684016a70 con 0x7f469413e2a0 2024-09-16T03:07:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.088+0000 7f468b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.088+0000 7f468b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:12.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.336+0000 7f469960b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:07:12.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.336+0000 7f469960b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:12.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.336+0000 7f469960b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:12.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.336+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4678005380 con 0x7f469413e2a0 2024-09-16T03:07:12.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.336+0000 7f468b7fe640 1 -- 172.21.15.45:0/3778209596 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2241153981) 0x7f468401fcd0 con 0x7f469413e2a0 2024-09-16T03:07:12.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.336+0000 7f468b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:12.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.336+0000 7f468b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:12.341 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:07:12.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 10 librados: watch_flush enter 2024-09-16T03:07:12.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 10 librados: watch_flush exit 2024-09-16T03:07:12.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 20 client.28442.objecter shutdown clearing up homeless session... 2024-09-16T03:07:12.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 10 client.28442.objecter successfully canceled tick 2024-09-16T03:07:12.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 >> v1:172.21.15.46:6812/149360280 conn(0x7f467403e060 legacy=0x7f4674040520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 10 monclient: shutdown 2024-09-16T03:07:12.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:12.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 >> v1:172.21.15.45:6790/0 conn(0x7f469413e2a0 legacy=0x7f46941d4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 shutdown_connections 2024-09-16T03:07:12.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 >> 172.21.15.45:0/3778209596 conn(0x7f46940aa110 msgr2=0x7f46940a0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:12.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 shutdown_connections 2024-09-16T03:07:12.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 1 -- 172.21.15.45:0/3778209596 wait complete. 2024-09-16T03:07:12.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.340+0000 7f469960b640 1 librados: shutdown 2024-09-16T03:07:12.360 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:07:12.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:07:12.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:07:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient: get_monmap_and_config 2024-09-16T03:07:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient: build_initial_monmap 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient: monmap: 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:12.488319+0000 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:12.488319+0000 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:12.490 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 Processor -- start 2024-09-16T03:07:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 -- start start 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient: init 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient(hunting): picked mon.noname-b con 0x7f42b00a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient(hunting): picked mon.noname-a con 0x7f42b00a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient(hunting): picked mon.noname-c con 0x7f42b0073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b009d9e0 con 0x7f42b00a5810 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b009dc20 con 0x7f42b0073d40 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b0073360 con 0x7f42b00a4d20 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient(hunting): authenticate will time out at 1123.137733s 2024-09-16T03:07:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b4f3d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f42b00a5810 0x7f42b013cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33608/0 (socket says 172.21.15.45:33608) 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b4f3d640 1 -- 172.21.15.45:0/118567431 learned_addr learned my addr 172.21.15.45:0/118567431 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b573e640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f42b00a4d20 0x7f42b00a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35544/0 (socket says 172.21.15.45:35544) 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844949966 0 0) 0x7f42b0073360 con 0x7f42b00a4d20 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient(hunting): my global_id is 28445 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4298003530 con 0x7f42b00a4d20 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1003420834 0 0) 0x7f42b009d9e0 con 0x7f42b00a5810 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient(hunting): my global_id is 94146 2024-09-16T03:07:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b0073360 con 0x7f42b00a5810 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4051476324 0 0) 0x7f4298003530 con 0x7f42b00a4d20 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 >> v1:172.21.15.46:6789/0 conn(0x7f42b0073d40 legacy=0x7f42b0140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 >> v1:172.21.15.45:6789/0 conn(0x7f42b00a5810 legacy=0x7f42b013cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: _finish_hunting 0 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 monclient: found mon.noname-b 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b009cf30 con 0x7f42b00a4d20 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: _finish_auth 0 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: _check_auth_tickets 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f42a8001950 con 0x7f42b00a4d20 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: dump: 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.497 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 5 monclient: authenticate success, global_id 28445 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42a8001b70 con 0x7f42b00a4d20 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 1 -- 172.21.15.45:0/118567431 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f42a8002000 con 0x7f42b00a4d20 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42affff640 10 monclient: dump: 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 4 set_mon_vals no callback set 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient: shutdown 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 -- 172.21.15.45:0/118567431 >> v1:172.21.15.45:6790/0 conn(0x7f42b00a4d20 legacy=0x7f42b00a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 -- 172.21.15.45:0/118567431 shutdown_connections 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 -- 172.21.15.45:0/118567431 >> 172.21.15.45:0/118567431 conn(0x7f42b00aa720 msgr2=0x7f42b00aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 -- 172.21.15.45:0/118567431 shutdown_connections 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 1 -- 172.21.15.45:0/118567431 wait complete. 2024-09-16T03:07:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.488+0000 7f42b79a2640 10 monclient: build_initial_monmap 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient: monmap: 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 librados: starting msgr at 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 librados: starting objecter 2024-09-16T03:07:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 Processor -- start 2024-09-16T03:07:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 -- start start 2024-09-16T03:07:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 librados: setting wanted keys 2024-09-16T03:07:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 librados: calling monclient init 2024-09-16T03:07:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient: init 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient(hunting): picked mon.noname-b con 0x7f42b0073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient(hunting): picked mon.noname-c con 0x7f42b00a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient(hunting): picked mon.noname-a con 0x7f42b00a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b01405e0 con 0x7f42b00a5810 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b01408c0 con 0x7f42b0073d40 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b00a1d00 con 0x7f42b00a4d20 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient(hunting): authenticate will time out at 1123.141053s 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b4f3d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f42b00a4d20 0x7f42b01d2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35546/0 (socket says 172.21.15.45:35546) 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b4f3d640 1 -- 172.21.15.45:0/2414986593 learned_addr learned my addr 172.21.15.45:0/2414986593 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 client.?.objecter ms_handle_connect 0x7f42b00a4d20 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 client.?.objecter ms_handle_connect 0x7f42b00a5810 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3079905395 0 0) 0x7f42b00a1d00 con 0x7f42b00a4d20 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): my global_id is 28448 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f429c004ec0 con 0x7f42b00a4d20 2024-09-16T03:07:12.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 client.?.objecter ms_handle_connect 0x7f42b0073d40 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253494102 0 0) 0x7f42b01405e0 con 0x7f42b00a5810 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): my global_id is 94149 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b00a1d00 con 0x7f42b00a5810 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3801433102 0 0) 0x7f429c004ec0 con 0x7f42b00a4d20 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42b01405e0 con 0x7f42b00a4d20 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 783690061 0 0) 0x7f42b01408c0 con 0x7f42b0073d40 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): my global_id is 22198 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f429c004ec0 con 0x7f42b0073d40 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f42a40012e0 con 0x7f42b00a4d20 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): dump: 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2840635890 0 0) 0x7f42b00a1d00 con 0x7f42b00a5810 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42b01408c0 con 0x7f42b00a5810 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f42a0001840 con 0x7f42b00a5810 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): dump: 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.511 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2151205085 0 0) 0x7f42b01405e0 con 0x7f42b00a4d20 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 >> v1:172.21.15.46:6789/0 conn(0x7f42b0073d40 legacy=0x7f42b01cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 >> v1:172.21.15.45:6789/0 conn(0x7f42b00a5810 legacy=0x7f42b01d5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: _finish_hunting 0 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 monclient: found mon.c 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:12.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b00a1510 con 0x7f42b00a4d20 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: _finish_auth 0 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: _check_auth_tickets 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 5 monclient: authenticate success, global_id 28448 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient: _renew_subs 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42b0140300 con 0x7f42b00a4d20 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 client.28448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient: _renew_subs 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f42b013dd50 con 0x7f42b00a4d20 2024-09-16T03:07:12.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 librados: init done 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42a4001620 con 0x7f42b00a4d20 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f42a4004c40 con 0x7f42b00a4d20 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: dump: 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.514 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42af7fe640 4 set_mon_vals no callback set 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f42900065d0 con 0x7f4290002870 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 client.28448.objecter ms_handle_connect 0x7f4290002870 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 client.28448.objecter resend_mon_ops 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f42a4010cb0 con 0x7f42b00a4d20 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(86..86 src has 1..86) ==== 7905+0+0 (unknown 2722104324 0 0) 0x7f42a404c220 con 0x7f42b00a4d20 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 client.28448.objecter ms_dispatch 0x7f42b0000ba0 osd_map(86..86 src has 1..86) 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 3 client.28448.objecter handle_osd_map got epochs [86,86] > 0 2024-09-16T03:07:12.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 3 client.28448.objecter handle_osd_map decoding full epoch 86 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 20 client.28448.objecter dump_active .. 0 homeless 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2522951730 0 0) 0x7f42900065d0 con 0x7f4290002870 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient(hunting): my global_id is 94152 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b01405e0 con 0x7f4290002870 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3597290523 0 0) 0x7f42b01405e0 con 0x7f4290002870 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42900065d0 con 0x7f4290002870 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f42a8001c40 con 0x7f4290002870 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42b5f3f640 10 monclient: get_auth_request con 0x7f429c03e220 auth_method 0 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 10 monclient: dump: 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.492+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3538385931 0 0) 0x7f42900065d0 con 0x7f4290002870 2024-09-16T03:07:12.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.496+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f429000a170 con 0x7f4290002870 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.496+0000 7f42adffb640 10 client.28448.objecter ms_handle_connect 0x7f429c03e220 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.496+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f42a8001aa0 con 0x7f4290002870 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.496+0000 7f42adffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.496+0000 7f42adffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.496+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 >> v1:172.21.15.45:6789/0 conn(0x7f4290002870 legacy=0x7f4290004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.496+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 >> v1:172.21.15.45:6789/0 conn(0x7f4290002870 legacy=0x7f4290004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42b79a2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4290009e80 con 0x7f4290009140 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 client.28448.objecter ms_handle_connect 0x7f4290009140 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 client.28448.objecter resend_mon_ops 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879903000 0 0) 0x7f4290009e80 con 0x7f4290009140 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 monclient(hunting): my global_id is 94155 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42900065d0 con 0x7f4290009140 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1890095224 0 0) 0x7f42900065d0 con 0x7f4290009140 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4290009e80 con 0x7f4290009140 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f42a8005390 con 0x7f4290009140 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 monclient: dump: 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4251192918 0 0) 0x7f4290009e80 con 0x7f4290009140 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f429000c420 con 0x7f4290009140 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3969 (unknown 0 0 2528184031) 0x7f42a80075d0 con 0x7f4290009140 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 >> v1:172.21.15.45:6789/0 conn(0x7f4290009140 legacy=0x7f42900095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42adffb640 1 -- 172.21.15.45:0/2414986593 >> v1:172.21.15.45:6789/0 conn(0x7f4290009140 legacy=0x7f42900095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:12.521 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:07:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:07:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 12, 2024-09-16T03:07:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 15483, 2024-09-16T03:07:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:07:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:07:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:07:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:07:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:07:12.532 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:07:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42b79a2640 10 librados: watch_flush enter 2024-09-16T03:07:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42b79a2640 10 librados: watch_flush exit 2024-09-16T03:07:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42b79a2640 20 client.28448.objecter shutdown clearing up homeless session... 2024-09-16T03:07:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42b79a2640 10 client.28448.objecter successfully canceled tick 2024-09-16T03:07:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.500+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 >> v1:172.21.15.46:6812/149360280 conn(0x7f429c03e220 legacy=0x7f429c0406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b79a2640 10 monclient: shutdown 2024-09-16T03:07:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b79a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 >> v1:172.21.15.45:6790/0 conn(0x7f42b00a4d20 legacy=0x7f42b01d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b5f3f640 1 -- 172.21.15.45:0/2414986593 reap_dead start 2024-09-16T03:07:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b5f3f640 1 -- 172.21.15.45:0/2414986593 reap_dead start 2024-09-16T03:07:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 shutdown_connections 2024-09-16T03:07:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 >> 172.21.15.45:0/2414986593 conn(0x7f42b00aa720 msgr2=0x7f42b00a40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 shutdown_connections 2024-09-16T03:07:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b79a2640 1 -- 172.21.15.45:0/2414986593 wait complete. 2024-09-16T03:07:12.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.504+0000 7f42b79a2640 1 librados: shutdown 2024-09-16T03:07:12.535 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:07:12.535 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:07:12.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:07:12.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient: get_monmap_and_config 2024-09-16T03:07:12.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient: build_initial_monmap 2024-09-16T03:07:12.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient: monmap: 2024-09-16T03:07:12.658 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:12.659 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.659 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:12.656985+0000 2024-09-16T03:07:12.659 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:12.656985+0000 2024-09-16T03:07:12.659 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:12.659 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.659 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:12.659 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:12.659 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:12.659 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 1 Processor -- start 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 1 -- start start 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient: init 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient(hunting): picked mon.noname-b con 0x7f2da409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient(hunting): picked mon.noname-c con 0x7f2da409b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient(hunting): picked mon.noname-a con 0x7f2da413b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da409cc20 con 0x7f2da413b000 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da409d820 con 0x7f2da409b250 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da409b6a0 con 0x7f2da409b8e0 2024-09-16T03:07:12.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient(hunting): authenticate will time out at 1123.306357s 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da2ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2da409b8e0 0x7f2da409bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35550/0 (socket says 172.21.15.45:35550) 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da37fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2da413b000 0x7f2da413d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33666/0 (socket says 172.21.15.45:33666) 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da2ffd640 1 -- 172.21.15.45:0/742260207 learned_addr learned my addr 172.21.15.45:0/742260207 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384045088 0 0) 0x7f2da409b6a0 con 0x7f2da409b8e0 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): my global_id is 28451 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d90005740 con 0x7f2da409b8e0 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4096850058 0 0) 0x7f2da409cc20 con 0x7f2da413b000 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): my global_id is 94158 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2da409b6a0 con 0x7f2da413b000 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 983676096 0 0) 0x7f2da409d820 con 0x7f2da409b250 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient(hunting): my global_id is 22201 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2da409cc20 con 0x7f2da409b250 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3248512050 0 0) 0x7f2da409b6a0 con 0x7f2da413b000 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 >> v1:172.21.15.45:6790/0 conn(0x7f2da409b8e0 legacy=0x7f2da409bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 >> v1:172.21.15.46:6789/0 conn(0x7f2da409b250 legacy=0x7f2da41398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:07:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 monclient: found mon.noname-a 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da40a21f0 con 0x7f2da413b000 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d94001a00 con 0x7f2da413b000 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 5 monclient: authenticate success, global_id 94158 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:12.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: dump: 2024-09-16T03:07:12.672 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.672 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.672 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.672 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.672 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.672 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.672 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.673 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.673 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.673 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:12.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d94001ba0 con 0x7f2da413b000 2024-09-16T03:07:12.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:12.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 1 -- 172.21.15.45:0/742260207 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d94001f70 con 0x7f2da413b000 2024-09-16T03:07:12.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:12.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 4 set_mon_vals no callback set 2024-09-16T03:07:12.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:12.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da1ffb640 10 monclient: dump: 2024-09-16T03:07:12.674 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.674 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.674 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.674 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.674 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.675 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.675 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.675 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.675 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.675 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 10 monclient: shutdown 2024-09-16T03:07:12.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:12.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.656+0000 7f2da949d640 1 -- 172.21.15.45:0/742260207 >> v1:172.21.15.45:6789/0 conn(0x7f2da413b000 legacy=0x7f2da413d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- 172.21.15.45:0/742260207 shutdown_connections 2024-09-16T03:07:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- 172.21.15.45:0/742260207 >> 172.21.15.45:0/742260207 conn(0x7f2da40a2bd0 msgr2=0x7f2da40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- 172.21.15.45:0/742260207 shutdown_connections 2024-09-16T03:07:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- 172.21.15.45:0/742260207 wait complete. 2024-09-16T03:07:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient: build_initial_monmap 2024-09-16T03:07:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient: monmap: 2024-09-16T03:07:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:12.677 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:12.677 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:12.677 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.677 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:12.677 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:12.677 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:12.677 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 librados: starting msgr at 2024-09-16T03:07:12.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 librados: starting objecter 2024-09-16T03:07:12.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 Processor -- start 2024-09-16T03:07:12.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- start start 2024-09-16T03:07:12.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 librados: setting wanted keys 2024-09-16T03:07:12.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 librados: calling monclient init 2024-09-16T03:07:12.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient: init 2024-09-16T03:07:12.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:12.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:07:12.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient(hunting): picked mon.noname-a con 0x7f2da409b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient(hunting): picked mon.noname-c con 0x7f2da409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient(hunting): picked mon.noname-b con 0x7f2da413b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da40a5120 con 0x7f2da409b250 2024-09-16T03:07:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da413ab70 con 0x7f2da413b000 2024-09-16T03:07:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da409d2c0 con 0x7f2da409b8e0 2024-09-16T03:07:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient(hunting): authenticate will time out at 1123.309541s 2024-09-16T03:07:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da2ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2da409b250 0x7f2da41343f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33668/0 (socket says 172.21.15.45:33668) 2024-09-16T03:07:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da2ffd640 1 -- 172.21.15.45:0/2377654673 learned_addr learned my addr 172.21.15.45:0/2377654673 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 client.?.objecter ms_handle_connect 0x7f2da409b250 2024-09-16T03:07:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 client.?.objecter ms_handle_connect 0x7f2da409b8e0 2024-09-16T03:07:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4190241681 0 0) 0x7f2da40a5120 con 0x7f2da409b250 2024-09-16T03:07:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): my global_id is 94161 2024-09-16T03:07:12.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d84004ef0 con 0x7f2da409b250 2024-09-16T03:07:12.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824072720 0 0) 0x7f2da409d2c0 con 0x7f2da409b8e0 2024-09-16T03:07:12.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): my global_id is 28454 2024-09-16T03:07:12.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2da40a5120 con 0x7f2da409b8e0 2024-09-16T03:07:12.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 client.?.objecter ms_handle_connect 0x7f2da413b000 2024-09-16T03:07:12.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2676202533 0 0) 0x7f2d84004ef0 con 0x7f2da409b250 2024-09-16T03:07:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2da409d2c0 con 0x7f2da409b250 2024-09-16T03:07:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d9c0013c0 con 0x7f2da409b250 2024-09-16T03:07:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): dump: 2024-09-16T03:07:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.683 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.683 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.683 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.683 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.683 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.683 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.683 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:12.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3216902950 0 0) 0x7f2da40a5120 con 0x7f2da409b8e0 2024-09-16T03:07:12.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d84004ef0 con 0x7f2da409b8e0 2024-09-16T03:07:12.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d8c001840 con 0x7f2da409b8e0 2024-09-16T03:07:12.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:12.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:12.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): dump: 2024-09-16T03:07:12.684 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.684 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.684 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 356843935 0 0) 0x7f2da413ab70 con 0x7f2da413b000 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): my global_id is 22204 2024-09-16T03:07:12.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2da40a5120 con 0x7f2da413b000 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1231059226 0 0) 0x7f2da409d2c0 con 0x7f2da409b250 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 >> v1:172.21.15.45:6790/0 conn(0x7f2da409b8e0 legacy=0x7f2da41cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 >> v1:172.21.15.46:6789/0 conn(0x7f2da413b000 legacy=0x7f2da41d0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 monclient: found mon.a 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da4071e50 con 0x7f2da409b250 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 5 monclient: authenticate success, global_id 94161 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient: _renew_subs 2024-09-16T03:07:12.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2da4073ed0 con 0x7f2da409b250 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 client.94161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient: _renew_subs 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2da4134cd0 con 0x7f2da409b250 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 librados: init done 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d9c0047f0 con 0x7f2da409b250 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d9c004b40 con 0x7f2da409b250 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da17fa640 4 set_mon_vals no callback set 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient: dump: 2024-09-16T03:07:12.687 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2d880065d0 con 0x7f2d88002870 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 client.94161.objecter ms_handle_connect 0x7f2d88002870 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 client.94161.objecter resend_mon_ops 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 881360292 0 0) 0x7f2d880065d0 con 0x7f2d88002870 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 monclient(hunting): my global_id is 28457 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2da409d2c0 con 0x7f2d88002870 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f2d9c010c30 con 0x7f2da409b250 2024-09-16T03:07:12.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(86..86 src has 1..86) ==== 7905+0+0 (unknown 2722104324 0 0) 0x7f2d9c04c290 con 0x7f2da409b250 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 10 client.94161.objecter ms_dispatch 0x7f2da4000ba0 osd_map(86..86 src has 1..86) 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 3 client.94161.objecter handle_osd_map got epochs [86,86] > 0 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.660+0000 7f2d9b7fe640 3 client.94161.objecter handle_osd_map decoding full epoch 86 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 20 client.94161.objecter dump_active .. 0 homeless 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4017389769 0 0) 0x7f2da409d2c0 con 0x7f2d88002870 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d880065d0 con 0x7f2d88002870 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d8c003990 con 0x7f2d88002870 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 10 monclient: dump: 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.690 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2da37fe640 10 monclient: get_auth_request con 0x7f2d8403e1f0 auth_method 0 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4030493817 0 0) 0x7f2d880065d0 con 0x7f2d88002870 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d8800a170 con 0x7f2d88002870 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 10 client.94161.objecter ms_handle_connect 0x7f2d8403e1f0 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2d8c004f70 con 0x7f2d88002870 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 >> v1:172.21.15.45:6790/0 conn(0x7f2d88002870 legacy=0x7f2d88004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.664+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 >> v1:172.21.15.45:6790/0 conn(0x7f2d88002870 legacy=0x7f2d88004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:12.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2da949d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:07:12.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2d88009e60 con 0x7f2d88009120 2024-09-16T03:07:12.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 client.94161.objecter ms_handle_connect 0x7f2d88009120 2024-09-16T03:07:12.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 client.94161.objecter resend_mon_ops 2024-09-16T03:07:12.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1430228680 0 0) 0x7f2d88009e60 con 0x7f2d88009120 2024-09-16T03:07:12.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 monclient(hunting): my global_id is 28460 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d880065d0 con 0x7f2d88009120 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 409085588 0 0) 0x7f2d880065d0 con 0x7f2d88009120 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d88009e60 con 0x7f2d88009120 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2d8c005820 con 0x7f2d88009120 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 monclient: dump: 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.694 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.695 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1549892764 0 0) 0x7f2d88009e60 con 0x7f2d88009120 2024-09-16T03:07:12.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d8800c3b0 con 0x7f2d88009120 2024-09-16T03:07:12.695 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 12, 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 154520, 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.696 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:12.697 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:12.698 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:12.699 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:12.699 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:12.699 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.699 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:07:12.699 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:07:12.699 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:07:12.699 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:07:12.699 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:07:12.700 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.701 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.702 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:12.703 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:12.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.705 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.706 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:12.707 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.708 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:12.709 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:12.709 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:12.709 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.709 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.709 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:07:12.709 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:07:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 19479662) 0x7f2d8c005960 con 0x7f2d88009120 2024-09-16T03:07:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:07:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 >> v1:172.21.15.45:6790/0 conn(0x7f2d88009120 legacy=0x7f2d880095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.668+0000 7f2d9b7fe640 1 -- 172.21.15.45:0/2377654673 >> v1:172.21.15.45:6790/0 conn(0x7f2d88009120 legacy=0x7f2d880095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 10 librados: watch_flush enter 2024-09-16T03:07:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 10 librados: watch_flush exit 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 20 client.94161.objecter shutdown clearing up homeless session... 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 10 client.94161.objecter successfully canceled tick 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 >> v1:172.21.15.46:6812/149360280 conn(0x7f2d8403e1f0 legacy=0x7f2d840406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 10 monclient: shutdown 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 >> v1:172.21.15.45:6789/0 conn(0x7f2da409b250 legacy=0x7f2da41343f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da37fe640 1 -- 172.21.15.45:0/2377654673 reap_dead start 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da37fe640 1 -- 172.21.15.45:0/2377654673 reap_dead start 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 shutdown_connections 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 >> 172.21.15.45:0/2377654673 conn(0x7f2da40a2bd0 msgr2=0x7f2da409ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 shutdown_connections 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 1 -- 172.21.15.45:0/2377654673 wait complete. 2024-09-16T03:07:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.672+0000 7f2da949d640 1 librados: shutdown 2024-09-16T03:07:12.713 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:07:12.713 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:07:12.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:07:12.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient: get_monmap_and_config 2024-09-16T03:07:12.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient: build_initial_monmap 2024-09-16T03:07:12.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient: monmap: 2024-09-16T03:07:12.830 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:12.829103+0000 2024-09-16T03:07:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:12.829103+0000 2024-09-16T03:07:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:12.831 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 1 Processor -- start 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 1 -- start start 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient: init 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient(hunting): picked mon.noname-b con 0x7fdea0137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient(hunting): picked mon.noname-c con 0x7fdea0138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient(hunting): picked mon.noname-a con 0x7fdea013e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea00a4f90 con 0x7fdea013e2a0 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea00a4cb0 con 0x7fdea0138100 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea009dc70 con 0x7fdea0137d20 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient(hunting): authenticate will time out at 1123.478455s 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9effd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdea0137d20 0x7fdea00a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35582/0 (socket says 172.21.15.45:35582) 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9effd640 1 -- 172.21.15.45:0/3323826901 learned_addr learned my addr 172.21.15.45:0/3323826901 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1173790325 0 0) 0x7fdea00a4f90 con 0x7fdea013e2a0 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient(hunting): my global_id is 94164 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde88003530 con 0x7fdea013e2a0 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3171514085 0 0) 0x7fdea009dc70 con 0x7fdea0137d20 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient(hunting): my global_id is 28463 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea00a4f90 con 0x7fdea0137d20 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2779256250 0 0) 0x7fde88003530 con 0x7fdea013e2a0 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 >> v1:172.21.15.45:6790/0 conn(0x7fdea0137d20 legacy=0x7fdea00a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 >> v1:172.21.15.46:6789/0 conn(0x7fdea0138100 legacy=0x7fdea013cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 monclient: found mon.noname-a 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea009cc50 con 0x7fdea013e2a0 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: _finish_auth 0 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:07:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fde98001950 con 0x7fdea013e2a0 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: dump: 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.845 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 5 monclient: authenticate success, global_id 94164 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde98001b70 con 0x7fdea013e2a0 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 1 -- 172.21.15.45:0/3323826901 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fde98001fc0 con 0x7fdea013e2a0 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fde9dffb640 10 monclient: dump: 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.828+0000 7fdea5297640 4 set_mon_vals no callback set 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: shutdown 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- 172.21.15.45:0/3323826901 >> v1:172.21.15.45:6789/0 conn(0x7fdea013e2a0 legacy=0x7fdea0140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- 172.21.15.45:0/3323826901 shutdown_connections 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- 172.21.15.45:0/3323826901 >> 172.21.15.45:0/3323826901 conn(0x7fdea00aa110 msgr2=0x7fdea00aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- 172.21.15.45:0/3323826901 shutdown_connections 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- 172.21.15.45:0/3323826901 wait complete. 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: build_initial_monmap 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: monmap: 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 librados: starting msgr at 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 librados: starting objecter 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 Processor -- start 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- start start 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 librados: setting wanted keys 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 librados: calling monclient init 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: init 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:07:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient(hunting): picked mon.noname-a con 0x7fdea0137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient(hunting): picked mon.noname-c con 0x7fdea0138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient(hunting): picked mon.noname-b con 0x7fdea013e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea013c200 con 0x7fdea0137d20 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea0141ba0 con 0x7fdea013e2a0 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea0139770 con 0x7fdea0138100 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient(hunting): authenticate will time out at 1123.481780s 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde9effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdea0137d20 0x7fdea0138950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33686/0 (socket says 172.21.15.45:33686) 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde9effd640 1 -- 172.21.15.45:0/2922593520 learned_addr learned my addr 172.21.15.45:0/2922593520 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 client.?.objecter ms_handle_connect 0x7fdea0137d20 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 client.?.objecter ms_handle_connect 0x7fdea0138100 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969894390 0 0) 0x7fdea013c200 con 0x7fdea0137d20 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): my global_id is 94167 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde8c004ec0 con 0x7fdea0137d20 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 198872593 0 0) 0x7fdea0139770 con 0x7fdea0138100 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): my global_id is 28466 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea013c200 con 0x7fdea0138100 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 client.?.objecter ms_handle_connect 0x7fdea013e2a0 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:12.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3214201417 0 0) 0x7fde8c004ec0 con 0x7fdea0137d20 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdea0139770 con 0x7fdea0137d20 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fde94001920 con 0x7fdea0137d20 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): dump: 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3254216358 0 0) 0x7fdea013c200 con 0x7fdea0138100 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde8c004ec0 con 0x7fdea0138100 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fde900018b0 con 0x7fdea0138100 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): dump: 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1341828030 0 0) 0x7fdea0141ba0 con 0x7fdea013e2a0 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient(hunting): my global_id is 22210 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea013c200 con 0x7fdea013e2a0 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1209196849 0 0) 0x7fdea0139770 con 0x7fdea0137d20 2024-09-16T03:07:12.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 >> v1:172.21.15.45:6790/0 conn(0x7fdea0138100 legacy=0x7fdea0139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 >> v1:172.21.15.46:6789/0 conn(0x7fdea013e2a0 legacy=0x7fdea01d4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient: _finish_hunting 0 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 monclient: found mon.a 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea00a1300 con 0x7fdea0137d20 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient: _finish_auth 0 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient: _check_auth_tickets 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 5 monclient: authenticate success, global_id 94167 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: _renew_subs 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdea013dce0 con 0x7fdea0137d20 2024-09-16T03:07:12.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 client.94167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: _renew_subs 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdea0139e20 con 0x7fdea0137d20 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 librados: init done 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde940047f0 con 0x7fdea0137d20 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fde94004b40 con 0x7fdea0137d20 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 10 monclient: dump: 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.856 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde9cff9640 4 set_mon_vals no callback set 2024-09-16T03:07:12.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fde800065d0 con 0x7fde80002870 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fde94010bb0 con 0x7fdea0137d20 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.832+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(86..86 src has 1..86) ==== 7905+0+0 (unknown 2722104324 0 0) 0x7fde9404c0f0 con 0x7fdea0137d20 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 client.94167.objecter ms_dispatch 0x7fdea0000ba0 osd_map(86..86 src has 1..86) 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 3 client.94167.objecter handle_osd_map got epochs [86,86] > 0 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 3 client.94167.objecter handle_osd_map decoding full epoch 86 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 20 client.94167.objecter dump_active .. 0 homeless 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 client.94167.objecter ms_handle_connect 0x7fde80002870 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 client.94167.objecter resend_mon_ops 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde9f7fe640 10 monclient: get_auth_request con 0x7fde8c03e240 auth_method 0 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574842417 0 0) 0x7fde800065d0 con 0x7fde80002870 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 monclient(hunting): my global_id is 22213 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea0139770 con 0x7fde80002870 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 client.94167.objecter ms_handle_connect 0x7fde8c03e240 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 561564957 0 0) 0x7fdea0139770 con 0x7fde80002870 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde800065d0 con 0x7fde80002870 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fde90003a50 con 0x7fde80002870 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 monclient: dump: 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.859 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 743037212 0 0) 0x7fde800065d0 con 0x7fde80002870 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde8000a170 con 0x7fde80002870 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fde900050b0 con 0x7fde80002870 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 >> v1:172.21.15.46:6789/0 conn(0x7fde80002870 legacy=0x7fde80004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.836+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 >> v1:172.21.15.46:6789/0 conn(0x7fde80002870 legacy=0x7fde80004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:07:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fde80009e60 con 0x7fde80009120 2024-09-16T03:07:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 client.94167.objecter ms_handle_connect 0x7fde80009120 2024-09-16T03:07:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 client.94167.objecter resend_mon_ops 2024-09-16T03:07:12.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523196167 0 0) 0x7fde80009e60 con 0x7fde80009120 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 monclient(hunting): my global_id is 22216 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde800065d0 con 0x7fde80009120 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2474020136 0 0) 0x7fde800065d0 con 0x7fde80009120 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde80009e60 con 0x7fde80009120 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fde90005900 con 0x7fde80009120 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 monclient: dump: 2024-09-16T03:07:12.863 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:12.864 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:12.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3393349424 0 0) 0x7fde80009e60 con 0x7fde80009120 2024-09-16T03:07:12.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fde8000c3b0 con 0x7fde80009120 2024-09-16T03:07:12.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 3376518395) 0x7fde90005a40 con 0x7fde80009120 2024-09-16T03:07:12.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:07:12.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:12.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 >> v1:172.21.15.46:6789/0 conn(0x7fde80009120 legacy=0x7fde800095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde877fe640 1 -- 172.21.15.45:0/2922593520 >> v1:172.21.15.46:6789/0 conn(0x7fde80009120 legacy=0x7fde800095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:12.865 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 12, 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 154649, 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:07:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:07:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:07:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:07:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:07:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:07:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:07:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 7 2024-09-16T03:07:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:07:12.879 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:07:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 10 librados: watch_flush enter 2024-09-16T03:07:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 10 librados: watch_flush exit 2024-09-16T03:07:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 20 client.94167.objecter shutdown clearing up homeless session... 2024-09-16T03:07:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 10 client.94167.objecter successfully canceled tick 2024-09-16T03:07:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 >> v1:172.21.15.46:6812/149360280 conn(0x7fde8c03e240 legacy=0x7fde8c040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 10 monclient: shutdown 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 >> v1:172.21.15.45:6789/0 conn(0x7fdea0137d20 legacy=0x7fdea0138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde9f7fe640 1 -- 172.21.15.45:0/2922593520 reap_dead start 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fde9f7fe640 1 -- 172.21.15.45:0/2922593520 reap_dead start 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 shutdown_connections 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 >> 172.21.15.45:0/2922593520 conn(0x7fdea00aa110 msgr2=0x7fdea00a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 shutdown_connections 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 1 -- 172.21.15.45:0/2922593520 wait complete. 2024-09-16T03:07:12.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:12.844+0000 7fdea5297640 1 librados: shutdown 2024-09-16T03:07:12.883 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:07:12.883 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:07:12.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:07:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 10 monclient: get_monmap_and_config 2024-09-16T03:07:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 10 monclient: build_initial_monmap 2024-09-16T03:07:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 10 monclient: monmap: 2024-09-16T03:07:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:13.009 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:13.006516+0000 2024-09-16T03:07:13.009 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:13.006516+0000 2024-09-16T03:07:13.009 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:13.009 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:13.009 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:13.009 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:13.010 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:13.010 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:13.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 1 Processor -- start 2024-09-16T03:07:13.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 1 -- start start 2024-09-16T03:07:13.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 10 monclient: init 2024-09-16T03:07:13.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:13.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:07:13.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 10 monclient(hunting): picked mon.noname-a con 0x7f1fdc135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:13.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 10 monclient(hunting): picked mon.noname-b con 0x7f1fdc09fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:13.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 10 monclient(hunting): picked mon.noname-c con 0x7f1fdc13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:13.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fdc09f7e0 con 0x7f1fdc135950 2024-09-16T03:07:13.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.004+0000 7f1fe3945640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fdc09b160 con 0x7f1fdc13c010 2024-09-16T03:07:13.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fdc09b440 con 0x7f1fdc09fc40 2024-09-16T03:07:13.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:13.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient(hunting): authenticate will time out at 1123.656115s 2024-09-16T03:07:13.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe16e1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1fdc135950 0x7f1fdc135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33694/0 (socket says 172.21.15.45:33694) 2024-09-16T03:07:13.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe0ee0640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1fdc09fc40 0x7f1fdc13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35612/0 (socket says 172.21.15.45:35612) 2024-09-16T03:07:13.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe16e1640 1 -- 172.21.15.45:0/1532453411 learned_addr learned my addr 172.21.15.45:0/1532453411 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:13.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281217032 0 0) 0x7f1fdc09f7e0 con 0x7f1fdc135950 2024-09-16T03:07:13.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:13.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:13.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:13.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient(hunting): my global_id is 94170 2024-09-16T03:07:13.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc8003530 con 0x7f1fdc135950 2024-09-16T03:07:13.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2996191320 0 0) 0x7f1fdc09b440 con 0x7f1fdc09fc40 2024-09-16T03:07:13.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:13.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:13.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:13.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient(hunting): my global_id is 28469 2024-09-16T03:07:13.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fdc09f7e0 con 0x7f1fdc09fc40 2024-09-16T03:07:13.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4187834751 0 0) 0x7f1fc8003530 con 0x7f1fdc135950 2024-09-16T03:07:13.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 >> v1:172.21.15.45:6790/0 conn(0x7f1fdc09fc40 legacy=0x7f1fdc13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:13.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 >> v1:172.21.15.46:6789/0 conn(0x7f1fdc13c010 legacy=0x7f1fdc13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:13.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: _finish_hunting 0 2024-09-16T03:07:13.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 monclient: found mon.noname-a 2024-09-16T03:07:13.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:13.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:07:13.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fdc136440 con 0x7f1fdc135950 2024-09-16T03:07:13.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: _finish_auth 0 2024-09-16T03:07:13.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: _check_auth_tickets 2024-09-16T03:07:13.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:13.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fd00015d0 con 0x7f1fdc135950 2024-09-16T03:07:13.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:13.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:13.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: dump: 2024-09-16T03:07:13.031 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:13.031 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:13.031 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.031 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.032 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:13.032 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:13.032 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:13.032 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:13.032 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:13.033 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:13.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 5 monclient: authenticate success, global_id 94170 2024-09-16T03:07:13.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:13.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fd00017f0 con 0x7f1fdc135950 2024-09-16T03:07:13.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:13.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 1 -- 172.21.15.45:0/1532453411 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fd0001c40 con 0x7f1fdc135950 2024-09-16T03:07:13.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:13.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:13.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcffff640 10 monclient: dump: 2024-09-16T03:07:13.034 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:13.034 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:13.034 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.035 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.035 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:13.035 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:13.035 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:13.035 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:13.036 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:13.036 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:13.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:13.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 4 set_mon_vals no callback set 2024-09-16T03:07:13.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient: shutdown 2024-09-16T03:07:13.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:13.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- 172.21.15.45:0/1532453411 >> v1:172.21.15.45:6789/0 conn(0x7f1fdc135950 legacy=0x7f1fdc135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:13.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- 172.21.15.45:0/1532453411 shutdown_connections 2024-09-16T03:07:13.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- 172.21.15.45:0/1532453411 >> 172.21.15.45:0/1532453411 conn(0x7f1fdc0a2bd0 msgr2=0x7f1fdc0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:13.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- 172.21.15.45:0/1532453411 shutdown_connections 2024-09-16T03:07:13.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- 172.21.15.45:0/1532453411 wait complete. 2024-09-16T03:07:13.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient: build_initial_monmap 2024-09-16T03:07:13.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient: monmap: 2024-09-16T03:07:13.038 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:13.039 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:13.039 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:13.039 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:13.039 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:13.039 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:13.040 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:13.040 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:13.040 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:13.040 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:13.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 librados: starting msgr at 2024-09-16T03:07:13.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 librados: starting objecter 2024-09-16T03:07:13.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 Processor -- start 2024-09-16T03:07:13.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- start start 2024-09-16T03:07:13.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 librados: setting wanted keys 2024-09-16T03:07:13.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 librados: calling monclient init 2024-09-16T03:07:13.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient: init 2024-09-16T03:07:13.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:13.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:07:13.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient(hunting): picked mon.noname-b con 0x7f1fdc09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:13.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient(hunting): picked mon.noname-c con 0x7f1fdc135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:13.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient(hunting): picked mon.noname-a con 0x7f1fdc13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:13.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fdc071320 con 0x7f1fdc13c010 2024-09-16T03:07:13.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fdc071600 con 0x7f1fdc09fc40 2024-09-16T03:07:13.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fdc070fd0 con 0x7f1fdc135950 2024-09-16T03:07:13.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:13.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe3945640 10 monclient(hunting): authenticate will time out at 1123.659159s 2024-09-16T03:07:13.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe0ee0640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1fdc135950 0x7f1fdc06ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35616/0 (socket says 172.21.15.45:35616) 2024-09-16T03:07:13.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fe0ee0640 1 -- 172.21.15.45:0/3986421210 learned_addr learned my addr 172.21.15.45:0/3986421210 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:13.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 client.?.objecter ms_handle_connect 0x7f1fdc135950 2024-09-16T03:07:13.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:13.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 client.?.objecter ms_handle_connect 0x7f1fdc13c010 2024-09-16T03:07:13.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:13.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221034982 0 0) 0x7f1fdc071320 con 0x7f1fdc13c010 2024-09-16T03:07:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): my global_id is 94173 2024-09-16T03:07:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc0004ec0 con 0x7f1fdc13c010 2024-09-16T03:07:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 client.?.objecter ms_handle_connect 0x7f1fdc09fc40 2024-09-16T03:07:13.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:13.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245982956 0 0) 0x7f1fdc070fd0 con 0x7f1fdc135950 2024-09-16T03:07:13.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:13.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:13.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): my global_id is 28472 2024-09-16T03:07:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fdc071320 con 0x7f1fdc135950 2024-09-16T03:07:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4171096223 0 0) 0x7f1fc0004ec0 con 0x7f1fdc13c010 2024-09-16T03:07:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fdc070fd0 con 0x7f1fdc13c010 2024-09-16T03:07:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145028484 0 0) 0x7f1fdc071600 con 0x7f1fdc09fc40 2024-09-16T03:07:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): my global_id is 22222 2024-09-16T03:07:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc0004ec0 con 0x7f1fdc09fc40 2024-09-16T03:07:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fd40018c0 con 0x7f1fdc13c010 2024-09-16T03:07:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): dump: 2024-09-16T03:07:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.050 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.050 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:13.050 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:13.050 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:13.050 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:13.051 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4107026343 0 0) 0x7f1fdc071320 con 0x7f1fdc135950 2024-09-16T03:07:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fdc071600 con 0x7f1fdc135950 2024-09-16T03:07:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fd8001b80 con 0x7f1fdc135950 2024-09-16T03:07:13.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:13.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:13.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.008+0000 7f1fcdffb640 10 monclient(hunting): dump: 2024-09-16T03:07:13.052 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:13.052 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:13.052 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:13.054 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 825566672 0 0) 0x7f1fdc070fd0 con 0x7f1fdc13c010 2024-09-16T03:07:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 >> v1:172.21.15.45:6790/0 conn(0x7f1fdc135950 legacy=0x7f1fdc06ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 >> v1:172.21.15.46:6789/0 conn(0x7f1fdc09fc40 legacy=0x7f1fdc06e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:13.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 monclient: _finish_hunting 0 2024-09-16T03:07:13.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 1 monclient: found mon.a 2024-09-16T03:07:13.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:13.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:13.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fdc0703c0 con 0x7f1fdc13c010 2024-09-16T03:07:13.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 monclient: _finish_auth 0 2024-09-16T03:07:13.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 monclient: _check_auth_tickets 2024-09-16T03:07:13.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:13.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 5 monclient: authenticate success, global_id 94173 2024-09-16T03:07:13.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 10 monclient: _renew_subs 2024-09-16T03:07:13.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:13.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1fdc06f430 con 0x7f1fdc13c010 2024-09-16T03:07:13.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 10 client.94173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:13.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 10 monclient: _renew_subs 2024-09-16T03:07:13.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:13.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1fdc1d62c0 con 0x7f1fdc13c010 2024-09-16T03:07:13.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 1 librados: init done 2024-09-16T03:07:13.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fd4003600 con 0x7f1fdc13c010 2024-09-16T03:07:13.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:13.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fd4003950 con 0x7f1fdc13c010 2024-09-16T03:07:13.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcf7fe640 4 set_mon_vals no callback set 2024-09-16T03:07:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 monclient: dump: 2024-09-16T03:07:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:13.061 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fbc005090 con 0x7f1fdc13c010 2024-09-16T03:07:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f1fd400f9c0 con 0x7f1fdc13c010 2024-09-16T03:07:13.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(86..86 src has 1..86) ==== 7905+0+0 (unknown 2722104324 0 0) 0x7f1fd404ae50 con 0x7f1fdc13c010 2024-09-16T03:07:13.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 client.94173.objecter ms_dispatch 0x7f1fdc000ba0 osd_map(86..86 src has 1..86) 2024-09-16T03:07:13.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 3 client.94173.objecter handle_osd_map got epochs [86,86] > 0 2024-09-16T03:07:13.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 3 client.94173.objecter handle_osd_map decoding full epoch 86 2024-09-16T03:07:13.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 20 client.94173.objecter dump_active .. 0 homeless 2024-09-16T03:07:13.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fe16e1640 10 monclient: get_auth_request con 0x7f1fc003e290 auth_method 0 2024-09-16T03:07:13.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.012+0000 7f1fcdffb640 10 client.94173.objecter ms_handle_connect 0x7f1fc003e290 2024-09-16T03:07:13.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.016+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1fd4015120 con 0x7f1fdc13c010 2024-09-16T03:07:13.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.016+0000 7f1fcdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:13.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.016+0000 7f1fcdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.292+0000 7f1fe3945640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:07:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.292+0000 7f1fe3945640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:07:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.292+0000 7f1fe3945640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.292+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1fbc005c50 con 0x7f1fdc13c010 2024-09-16T03:07:13.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.296+0000 7f1fcdffb640 1 -- 172.21.15.45:0/3986421210 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f1fd401e9d0 con 0x7f1fdc13c010 2024-09-16T03:07:13.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.296+0000 7f1fcdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:07:13.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.296+0000 7f1fcdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:13.300 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.296+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.296+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=14,logm=57} crc {auth=226894307,config=1097430493,health=2307676939,kv=3374818887,logm=2546912952}) 2024-09-16T03:07:13.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.296+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=11,kv=14,logm=57} crc {auth=2748390738,config=1097430493,health=668193703,kv=3374818887,logm=4014389861}) 2024-09-16T03:07:13.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.296+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.296+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=14,logm=57} crc {auth=226894307,config=1097430493,health=2307676939,kv=3374818887,logm=2546912952}) 2024-09-16T03:07:13.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.296+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=10,kv=14,logm=59} crc {auth=226894307,config=1097430493,health=1336710883,kv=235459132,logm=4039220996}) 2024-09-16T03:07:13.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 10 librados: watch_flush enter 2024-09-16T03:07:13.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 10 librados: watch_flush exit 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 20 client.94173.objecter shutdown clearing up homeless session... 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 10 client.94173.objecter successfully canceled tick 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 >> v1:172.21.15.46:6812/149360280 conn(0x7f1fc003e290 legacy=0x7f1fc0040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 10 monclient: shutdown 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 >> v1:172.21.15.45:6789/0 conn(0x7f1fdc13c010 legacy=0x7f1fdc1d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 shutdown_connections 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 >> 172.21.15.45:0/3986421210 conn(0x7f1fdc0a2bd0 msgr2=0x7f1fdc13af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 shutdown_connections 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 1 -- 172.21.15.45:0/3986421210 wait complete. 2024-09-16T03:07:13.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f1fe3945640 1 librados: shutdown 2024-09-16T03:07:13.304 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.304 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1703662659}) 2024-09-16T03:07:13.304 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=395555355}) 2024-09-16T03:07:13.304 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.304 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1703662659}) 2024-09-16T03:07:13.305 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=501188796}) 2024-09-16T03:07:13.305 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.305 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593500777}) 2024-09-16T03:07:13.305 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842804205}) 2024-09-16T03:07:13.306 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.300+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.306 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.304+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593500777}) 2024-09-16T03:07:13.306 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.304+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2718566279}) 2024-09-16T03:07:13.307 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.304+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.307 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.304+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516750642}) 2024-09-16T03:07:13.307 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.304+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=251824082}) 2024-09-16T03:07:13.308 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.304+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.308 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.304+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516750642}) 2024-09-16T03:07:13.308 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.304+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1360303091}) 2024-09-16T03:07:13.309 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.310 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244415470}) 2024-09-16T03:07:13.310 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692123884}) 2024-09-16T03:07:13.310 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.310 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244415470}) 2024-09-16T03:07:13.310 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363539412}) 2024-09-16T03:07:13.312 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.312 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983466610}) 2024-09-16T03:07:13.312 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838223102}) 2024-09-16T03:07:13.312 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.313 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983466610}) 2024-09-16T03:07:13.313 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.308+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2741517123}) 2024-09-16T03:07:13.314 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.314 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213439098}) 2024-09-16T03:07:13.314 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483025456}) 2024-09-16T03:07:13.314 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.314 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213439098}) 2024-09-16T03:07:13.314 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518572979}) 2024-09-16T03:07:13.316 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.316 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2376336007}) 2024-09-16T03:07:13.316 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=720050399}) 2024-09-16T03:07:13.316 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.316 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2376336007}) 2024-09-16T03:07:13.316 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.312+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=50082500}) 2024-09-16T03:07:13.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:07:13.318 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.318 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140851995}) 2024-09-16T03:07:13.319 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093297844}) 2024-09-16T03:07:13.319 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.319 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140851995}) 2024-09-16T03:07:13.319 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2173995254}) 2024-09-16T03:07:13.320 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.320 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1454553052}) 2024-09-16T03:07:13.320 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515898055}) 2024-09-16T03:07:13.320 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.321 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1454553052}) 2024-09-16T03:07:13.321 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.316+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2639059774}) 2024-09-16T03:07:13.322 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.322 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43301028}) 2024-09-16T03:07:13.322 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3908390786}) 2024-09-16T03:07:13.323 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.323 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43301028}) 2024-09-16T03:07:13.323 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=409586664}) 2024-09-16T03:07:13.325 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.325 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137899}) 2024-09-16T03:07:13.325 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3046991288}) 2024-09-16T03:07:13.325 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.325 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137899}) 2024-09-16T03:07:13.325 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.320+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2080279241}) 2024-09-16T03:07:13.326 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.326 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021395304}) 2024-09-16T03:07:13.326 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2716062491}) 2024-09-16T03:07:13.326 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.327 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021395304}) 2024-09-16T03:07:13.327 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=968730390}) 2024-09-16T03:07:13.329 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.329 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=18,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=15} crc {logm=2595321791,mdsmap=2287870078,mgr=3069726678,mgr_metadata=127159686,mgrstat=744458612,mon_config_key=612628065,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=343107189,osd_snap=1648405992,osdmap=1591390244}) 2024-09-16T03:07:13.329 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=18,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14,osdmap=12} crc {logm=449244389,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=744458612,mon_config_key=3258009752,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=343107189,osd_snap=3790817843,osdmap=1386777035}) 2024-09-16T03:07:13.329 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.329 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=18,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=15} crc {logm=2595321791,mdsmap=2287870078,mgr=3069726678,mgr_metadata=127159686,mgrstat=744458612,mon_config_key=612628065,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=343107189,osd_snap=1648405992,osdmap=1591390244}) 2024-09-16T03:07:13.330 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.324+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=18,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=18,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=14,osdmap=17} crc {logm=3902850282,mdsmap=2287870078,mgr=1407849713,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=3157363420,mon_config_key=1839808364,monmap=3372108098,nvmeofgw=598337686,osd_metadata=1913529591,osd_pg_creating=343107189,osd_snap=2731743119,osdmap=1819307520}) 2024-09-16T03:07:13.333 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.328+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.333 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.328+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2671610384}) 2024-09-16T03:07:13.333 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.328+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2551764379}) 2024-09-16T03:07:13.333 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.328+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.333 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.328+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2671610384}) 2024-09-16T03:07:13.333 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.328+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2879940588}) 2024-09-16T03:07:13.341 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.340+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.341 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.340+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=2454516876}) 2024-09-16T03:07:13.341 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.340+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=32} crc {osdmap=1012811735}) 2024-09-16T03:07:13.342 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.340+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:07:13.342 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.340+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=2454516876}) 2024-09-16T03:07:13.342 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:07:13.340+0000 7f474216f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=31} crc {osdmap=616466629}) 2024-09-16T03:07:14.318 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:07:14.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:07:14.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.448+0000 7f70f2eac640 10 monclient: get_monmap_and_config 2024-09-16T03:07:14.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.448+0000 7f70f2eac640 10 monclient: build_initial_monmap 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.448+0000 7f70f2eac640 10 monclient: monmap: 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:14.450848+0000 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:14.450848+0000 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.448+0000 7f70f2eac640 1 Processor -- start 2024-09-16T03:07:14.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.448+0000 7f70f2eac640 1 -- start start 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.448+0000 7f70f2eac640 10 monclient: init 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient(hunting): picked mon.noname-a con 0x7f70ec135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient(hunting): picked mon.noname-c con 0x7f70ec09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient(hunting): picked mon.noname-b con 0x7f70ec13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ec09f7e0 con 0x7f70ec135950 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ec09b160 con 0x7f70ec09fc40 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ec09b440 con 0x7f70ec13c010 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient(hunting): authenticate will time out at 1125.100733s 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f1449640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f70ec13c010 0x7f70ec13e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35624/0 (socket says 172.21.15.45:35624) 2024-09-16T03:07:14.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f0c48640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f70ec135950 0x7f70ec135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33722/0 (socket says 172.21.15.45:33722) 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f1449640 1 -- 172.21.15.45:0/1812873205 learned_addr learned my addr 172.21.15.45:0/1812873205 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3159636188 0 0) 0x7f70ec09b440 con 0x7f70ec13c010 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient(hunting): my global_id is 28487 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70dc003530 con 0x7f70ec13c010 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383722346 0 0) 0x7f70ec09f7e0 con 0x7f70ec135950 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient(hunting): my global_id is 94185 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70ec09b440 con 0x7f70ec135950 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1263183993 0 0) 0x7f70dc003530 con 0x7f70ec13c010 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 >> v1:172.21.15.46:6789/0 conn(0x7f70ec09fc40 legacy=0x7f70ec13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 >> v1:172.21.15.45:6789/0 conn(0x7f70ec135950 legacy=0x7f70ec135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 monclient: found mon.noname-b 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ec136440 con 0x7f70ec13c010 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:07:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f70e4001a20 con 0x7f70ec13c010 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 5 monclient: authenticate success, global_id 28487 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: dump: 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.482 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70e4001c40 con 0x7f70ec13c010 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 4 set_mon_vals no callback set 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 1 -- 172.21.15.45:0/1812873205 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f70e4002090 con 0x7f70ec13c010 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:14.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70eb7fe640 10 monclient: dump: 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient: shutdown 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:14.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 -- 172.21.15.45:0/1812873205 >> v1:172.21.15.45:6790/0 conn(0x7f70ec13c010 legacy=0x7f70ec13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 -- 172.21.15.45:0/1812873205 shutdown_connections 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 -- 172.21.15.45:0/1812873205 >> 172.21.15.45:0/1812873205 conn(0x7f70ec0a2bd0 msgr2=0x7f70ec0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 -- 172.21.15.45:0/1812873205 shutdown_connections 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 -- 172.21.15.45:0/1812873205 wait complete. 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient: build_initial_monmap 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient: monmap: 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:14.485 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 librados: starting msgr at 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 librados: starting objecter 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 Processor -- start 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 -- start start 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 librados: setting wanted keys 2024-09-16T03:07:14.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 1 librados: calling monclient init 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient: init 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.452+0000 7f70f2eac640 10 monclient(hunting): picked mon.noname-a con 0x7f70ec09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient(hunting): picked mon.noname-c con 0x7f70ec135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient(hunting): picked mon.noname-b con 0x7f70ec13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ec09f020 con 0x7f70ec09fc40 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ec09e7c0 con 0x7f70ec13c010 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ec13bc40 con 0x7f70ec135950 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:14.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient(hunting): authenticate will time out at 1125.104078s 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f0c48640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f70ec09fc40 0x7f70ec099290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33732/0 (socket says 172.21.15.45:33732) 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f0c48640 1 -- 172.21.15.45:0/1254759687 learned_addr learned my addr 172.21.15.45:0/1254759687 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 client.?.objecter ms_handle_connect 0x7f70ec09fc40 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 client.?.objecter ms_handle_connect 0x7f70ec135950 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2024504021 0 0) 0x7f70ec13bc40 con 0x7f70ec135950 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): my global_id is 28490 2024-09-16T03:07:14.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70d0004ec0 con 0x7f70ec135950 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806139952 0 0) 0x7f70ec09f020 con 0x7f70ec09fc40 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): my global_id is 94188 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70ec13bc40 con 0x7f70ec09fc40 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 client.?.objecter ms_handle_connect 0x7f70ec13c010 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2075055953 0 0) 0x7f70d0004ec0 con 0x7f70ec135950 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70ec09f020 con 0x7f70ec135950 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3667254517 0 0) 0x7f70ec13bc40 con 0x7f70ec09fc40 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70d0004ec0 con 0x7f70ec09fc40 2024-09-16T03:07:14.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f70d80018c0 con 0x7f70ec135950 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): dump: 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.490 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f70e00018c0 con 0x7f70ec09fc40 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): dump: 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.491 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 536986248 0 0) 0x7f70ec09e7c0 con 0x7f70ec13c010 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient(hunting): my global_id is 22240 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70ec13bc40 con 0x7f70ec13c010 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 506067889 0 0) 0x7f70d0004ec0 con 0x7f70ec09fc40 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 >> v1:172.21.15.45:6790/0 conn(0x7f70ec135950 legacy=0x7f70ec1d1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 >> v1:172.21.15.46:6789/0 conn(0x7f70ec13c010 legacy=0x7f70ec1d56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient: _finish_hunting 0 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 monclient: found mon.a 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ec09d2c0 con 0x7f70ec09fc40 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient: _finish_auth 0 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient: _check_auth_tickets 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 5 monclient: authenticate success, global_id 94188 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient: _renew_subs 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70ec09ed40 con 0x7f70ec09fc40 2024-09-16T03:07:14.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 client.94188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient: _renew_subs 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f70ec0737d0 con 0x7f70ec09fc40 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 1 librados: init done 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70e00047f0 con 0x7f70ec09fc40 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f70e0004c40 con 0x7f70ec09fc40 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70eaffd640 4 set_mon_vals no callback set 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 monclient: dump: 2024-09-16T03:07:14.494 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:14.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70cc005090 con 0x7f70ec09fc40 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f70e0004e60 con 0x7f70ec09fc40 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(88..88 src has 1..88) ==== 7967+0+0 (unknown 1418569008 0 0) 0x7f70e004c0b0 con 0x7f70ec09fc40 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 client.94188.objecter ms_dispatch 0x7f70ec000ba0 osd_map(88..88 src has 1..88) 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 3 client.94188.objecter handle_osd_map got epochs [88,88] > 0 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 3 client.94188.objecter handle_osd_map decoding full epoch 88 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 20 client.94188.objecter dump_active .. 0 homeless 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70ebfff640 10 monclient: get_auth_request con 0x7f70d003e1e0 auth_method 0 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.456+0000 7f70e97fa640 10 client.94188.objecter ms_handle_connect 0x7f70d003e1e0 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.460+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f70e00169e0 con 0x7f70ec09fc40 2024-09-16T03:07:14.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.460+0000 7f70e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:14.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.460+0000 7f70e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:07:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f70cc005380 con 0x7f70ec09fc40 2024-09-16T03:07:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70e97fa640 1 -- 172.21.15.45:0/1254759687 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 1624212498) 0x7f70e00162e0 con 0x7f70ec09fc40 2024-09-16T03:07:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:14.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70e97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:14.703 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 10 librados: watch_flush enter 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 10 librados: watch_flush exit 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 20 client.94188.objecter shutdown clearing up homeless session... 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 10 client.94188.objecter successfully canceled tick 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 >> v1:172.21.15.46:6812/149360280 conn(0x7f70d003e1e0 legacy=0x7f70d00406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 10 monclient: shutdown 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 >> v1:172.21.15.45:6789/0 conn(0x7f70ec09fc40 legacy=0x7f70ec099290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 shutdown_connections 2024-09-16T03:07:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 >> 172.21.15.45:0/1254759687 conn(0x7f70ec0a2bd0 msgr2=0x7f70ec13ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:14.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 shutdown_connections 2024-09-16T03:07:14.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 1 -- 172.21.15.45:0/1254759687 wait complete. 2024-09-16T03:07:14.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.700+0000 7f70f2eac640 1 librados: shutdown 2024-09-16T03:07:14.721 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:07:14.721 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:07:14.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:07:14.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:07:14.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient: get_monmap_and_config 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient: build_initial_monmap 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient: monmap: 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:14.845365+0000 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:14.845365+0000 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:14.847 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:14.848 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 1 Processor -- start 2024-09-16T03:07:14.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 1 -- start start 2024-09-16T03:07:14.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient: init 2024-09-16T03:07:14.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f8ecc135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f8ecc09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f8ecc13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ecc09f7e0 con 0x7f8ecc09fc40 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ecc09b160 con 0x7f8ecc13c010 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ecc09b440 con 0x7f8ecc135950 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient(hunting): authenticate will time out at 1125.494931s 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed0a36640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8ecc135950 0x7f8ecc135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35642/0 (socket says 172.21.15.45:35642) 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed0a36640 1 -- 172.21.15.45:0/1579432783 learned_addr learned my addr 172.21.15.45:0/1579432783 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475506673 0 0) 0x7f8ecc09b440 con 0x7f8ecc135950 2024-09-16T03:07:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient(hunting): my global_id is 28493 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb4003530 con 0x7f8ecc135950 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330256684 0 0) 0x7f8ecc09f7e0 con 0x7f8ecc09fc40 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient(hunting): my global_id is 94191 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ecc09b440 con 0x7f8ecc09fc40 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3322234861 0 0) 0x7f8eb4003530 con 0x7f8ecc135950 2024-09-16T03:07:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 >> v1:172.21.15.46:6789/0 conn(0x7f8ecc13c010 legacy=0x7f8ecc13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 >> v1:172.21.15.45:6789/0 conn(0x7f8ecc09fc40 legacy=0x7f8ecc13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 monclient: found mon.noname-b 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ecc136440 con 0x7f8ecc135950 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8ec4001950 con 0x7f8ecc135950 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:07:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: dump: 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 5 monclient: authenticate success, global_id 28493 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ec4001af0 con 0x7f8ecc135950 2024-09-16T03:07:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 1 -- 172.21.15.45:0/1579432783 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8ec4001f80 con 0x7f8ecc135950 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ecb7fe640 10 monclient: dump: 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.855 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.844+0000 7f8ed2c9a640 4 set_mon_vals no callback set 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: shutdown 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1579432783 >> v1:172.21.15.45:6790/0 conn(0x7f8ecc135950 legacy=0x7f8ecc135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1579432783 shutdown_connections 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1579432783 >> 172.21.15.45:0/1579432783 conn(0x7f8ecc0a2bd0 msgr2=0x7f8ecc0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1579432783 shutdown_connections 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1579432783 wait complete. 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: build_initial_monmap 2024-09-16T03:07:14.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: monmap: 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:14.857 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 librados: starting msgr at 2024-09-16T03:07:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 librados: starting objecter 2024-09-16T03:07:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 Processor -- start 2024-09-16T03:07:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- start start 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 librados: setting wanted keys 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 librados: calling monclient init 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: init 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f8ecc09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f8ecc135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f8ecc13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ecc13d4d0 con 0x7f8ecc135950 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ecc099d70 con 0x7f8ecc09fc40 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ecc13ef20 con 0x7f8ecc13c010 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient(hunting): authenticate will time out at 1125.498761s 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ecbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8ecc135950 0x7f8ecc1cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33736/0 (socket says 172.21.15.45:33736) 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ecbfff640 1 -- 172.21.15.45:0/1277485937 learned_addr learned my addr 172.21.15.45:0/1277485937 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 client.?.objecter ms_handle_connect 0x7f8ecc13c010 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 client.?.objecter ms_handle_connect 0x7f8ecc135950 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610820335 0 0) 0x7f8ecc13ef20 con 0x7f8ecc13c010 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): my global_id is 28496 2024-09-16T03:07:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb8004ec0 con 0x7f8ecc13c010 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 381459883 0 0) 0x7f8ecc13d4d0 con 0x7f8ecc135950 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): my global_id is 94194 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ecc13ef20 con 0x7f8ecc135950 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 client.?.objecter ms_handle_connect 0x7f8ecc09fc40 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2613822243 0 0) 0x7f8ecc13ef20 con 0x7f8ecc135950 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ecc13d4d0 con 0x7f8ecc135950 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8ec00012e0 con 0x7f8ecc135950 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): dump: 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1398421328 0 0) 0x7f8eb8004ec0 con 0x7f8ecc13c010 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ecc13ef20 con 0x7f8ecc13c010 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762568464 0 0) 0x7f8ecc099d70 con 0x7f8ecc09fc40 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): my global_id is 22246 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb8004ec0 con 0x7f8ecc09fc40 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8ebc001800 con 0x7f8ecc13c010 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient(hunting): dump: 2024-09-16T03:07:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4016352639 0 0) 0x7f8ecc13d4d0 con 0x7f8ecc135950 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 >> v1:172.21.15.45:6790/0 conn(0x7f8ecc13c010 legacy=0x7f8ecc1d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 >> v1:172.21.15.46:6789/0 conn(0x7f8ecc09fc40 legacy=0x7f8ecc13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient: _finish_hunting 0 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 monclient: found mon.a 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ecc13f200 con 0x7f8ecc135950 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient: _finish_auth 0 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient: _check_auth_tickets 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 5 monclient: authenticate success, global_id 94194 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: _renew_subs 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ecc0908e0 con 0x7f8ecc135950 2024-09-16T03:07:14.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 client.94194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: _renew_subs 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ecc1375f0 con 0x7f8ecc135950 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ed2c9a640 1 librados: init done 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ec0003600 con 0x7f8ecc135950 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8ec0003a10 con 0x7f8ecc135950 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ecaffd640 4 set_mon_vals no callback set 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.848+0000 7f8ec97fa640 10 monclient: dump: 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ed2c9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8eac0065d0 con 0x7f8eac002870 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f8ec0003c30 con 0x7f8ecc135950 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 client.94194.objecter ms_handle_connect 0x7f8eac002870 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 client.94194.objecter resend_mon_ops 2024-09-16T03:07:14.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(88..88 src has 1..88) ==== 7967+0+0 (unknown 1418569008 0 0) 0x7f8ec004c210 con 0x7f8ecc135950 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 client.94194.objecter ms_dispatch 0x7f8ecc000ba0 osd_map(88..88 src has 1..88) 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 3 client.94194.objecter handle_osd_map got epochs [88,88] > 0 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 3 client.94194.objecter handle_osd_map decoding full epoch 88 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 20 client.94194.objecter dump_active .. 0 homeless 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2898175540 0 0) 0x7f8eac0065d0 con 0x7f8eac002870 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 monclient(hunting): my global_id is 94197 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ecc13d4d0 con 0x7f8eac002870 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ed1237640 10 monclient: get_auth_request con 0x7f8eb803e1c0 auth_method 0 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3779547064 0 0) 0x7f8ecc13d4d0 con 0x7f8eac002870 2024-09-16T03:07:14.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8eac0065d0 con 0x7f8eac002870 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8ec4001bc0 con 0x7f8eac002870 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 monclient: dump: 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.873 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1430824851 0 0) 0x7f8eac0065d0 con 0x7f8eac002870 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8eac00a170 con 0x7f8eac002870 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 client.94194.objecter ms_handle_connect 0x7f8eb803e1c0 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8ec4001ba0 con 0x7f8eac002870 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 >> v1:172.21.15.45:6789/0 conn(0x7f8eac002870 legacy=0x7f8eac004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.852+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 >> v1:172.21.15.45:6789/0 conn(0x7f8eac002870 legacy=0x7f8eac004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8eac009e60 con 0x7f8eac009120 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 client.94194.objecter ms_handle_connect 0x7f8eac009120 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 client.94194.objecter resend_mon_ops 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675026506 0 0) 0x7f8eac009e60 con 0x7f8eac009120 2024-09-16T03:07:14.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 monclient(hunting): my global_id is 94200 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eac0065d0 con 0x7f8eac009120 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3503990837 0 0) 0x7f8eac0065d0 con 0x7f8eac009120 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8eac009e60 con 0x7f8eac009120 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8ec4004f00 con 0x7f8eac009120 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 monclient: dump: 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:14.875 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.876 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:14.876 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:14.876 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:14.876 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:14.876 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:14.876 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:14.876 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:14.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1053834214 0 0) 0x7f8eac009e60 con 0x7f8eac009120 2024-09-16T03:07:14.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8eac00c3b0 con 0x7f8eac009120 2024-09-16T03:07:14.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3969 (unknown 0 0 1733380682) 0x7f8ec4008f90 con 0x7f8eac009120 2024-09-16T03:07:14.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:07:14.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:14.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 >> v1:172.21.15.45:6789/0 conn(0x7f8eac009120 legacy=0x7f8eac0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ec97fa640 1 -- 172.21.15.45:0/1277485937 >> v1:172.21.15.45:6789/0 conn(0x7f8eac009120 legacy=0x7f8eac0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:14.877 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:07:14.877 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:14.877 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 12, 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 17843, 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:07:14.878 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:07:14.879 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:14.880 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:07:14.881 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:14.883 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:14.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:14.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:07:14.886 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:07:14.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:14.888 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:14.888 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:14.888 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:07:14.888 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:14.888 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:14.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:14.888 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:07:14.888 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:07:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 10 librados: watch_flush enter 2024-09-16T03:07:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 10 librados: watch_flush exit 2024-09-16T03:07:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 20 client.94194.objecter shutdown clearing up homeless session... 2024-09-16T03:07:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 10 client.94194.objecter successfully canceled tick 2024-09-16T03:07:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 >> v1:172.21.15.46:6812/149360280 conn(0x7f8eb803e1c0 legacy=0x7f8eb8040680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 10 monclient: shutdown 2024-09-16T03:07:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 >> v1:172.21.15.45:6789/0 conn(0x7f8ecc135950 legacy=0x7f8ecc1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed1237640 1 -- 172.21.15.45:0/1277485937 reap_dead start 2024-09-16T03:07:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed1237640 1 -- 172.21.15.45:0/1277485937 reap_dead start 2024-09-16T03:07:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 shutdown_connections 2024-09-16T03:07:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 >> 172.21.15.45:0/1277485937 conn(0x7f8ecc0a2bd0 msgr2=0x7f8ecc093400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 shutdown_connections 2024-09-16T03:07:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 1 -- 172.21.15.45:0/1277485937 wait complete. 2024-09-16T03:07:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:14.860+0000 7f8ed2c9a640 1 librados: shutdown 2024-09-16T03:07:14.891 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:07:14.891 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:07:14.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:07:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient: get_monmap_and_config 2024-09-16T03:07:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient: build_initial_monmap 2024-09-16T03:07:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient: monmap: 2024-09-16T03:07:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:15.005155+0000 2024-09-16T03:07:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:15.005155+0000 2024-09-16T03:07:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:15.007 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 1 Processor -- start 2024-09-16T03:07:15.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 1 -- start start 2024-09-16T03:07:15.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient: init 2024-09-16T03:07:15.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:15.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:07:15.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f08135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f0809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f0813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0809f7e0 con 0x7f2f0813c010 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0809b160 con 0x7f2f0809fc40 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0809b440 con 0x7f2f08135950 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient(hunting): authenticate will time out at 1125.654477s 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f077fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2f08135950 0x7f2f08135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35660/0 (socket says 172.21.15.45:35660) 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f077fe640 1 -- 172.21.15.45:0/1323041880 learned_addr learned my addr 172.21.15.45:0/1323041880 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1905475879 0 0) 0x7f2f0809b440 con 0x7f2f08135950 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient(hunting): my global_id is 28499 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ef0003530 con 0x7f2f08135950 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796860027 0 0) 0x7f2f0809f7e0 con 0x7f2f0813c010 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient(hunting): my global_id is 94203 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f0809b440 con 0x7f2f0813c010 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1783509617 0 0) 0x7f2f0809b440 con 0x7f2f0813c010 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 >> v1:172.21.15.45:6790/0 conn(0x7f2f08135950 legacy=0x7f2f08135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 >> v1:172.21.15.46:6789/0 conn(0x7f2f0809fc40 legacy=0x7f2f0813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: _finish_hunting 0 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 monclient: found mon.noname-a 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f08136440 con 0x7f2f0813c010 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: _finish_auth 0 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: _check_auth_tickets 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f00001a00 con 0x7f2f0813c010 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 5 monclient: authenticate success, global_id 94203 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: dump: 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f00001ba0 con 0x7f2f0813c010 2024-09-16T03:07:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 1 -- 172.21.15.45:0/1323041880 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f00002030 con 0x7f2f0813c010 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f067fc640 10 monclient: dump: 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 4 set_mon_vals no callback set 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 10 monclient: shutdown 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.004+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/1323041880 >> v1:172.21.15.45:6789/0 conn(0x7f2f0813c010 legacy=0x7f2f0813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/1323041880 shutdown_connections 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/1323041880 >> 172.21.15.45:0/1323041880 conn(0x7f2f080a2bd0 msgr2=0x7f2f080a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/1323041880 shutdown_connections 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/1323041880 wait complete. 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient: build_initial_monmap 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient: monmap: 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 librados: starting msgr at 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 librados: starting objecter 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 Processor -- start 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- start start 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 librados: setting wanted keys 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 librados: calling monclient init 2024-09-16T03:07:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient: init 2024-09-16T03:07:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:07:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f0809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f08135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f0813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f080708f0 con 0x7f2f0813c010 2024-09-16T03:07:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0813b8d0 con 0x7f2f08135950 2024-09-16T03:07:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f08134e70 con 0x7f2f0809fc40 2024-09-16T03:07:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient(hunting): authenticate will time out at 1125.657637s 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f077fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2f0809fc40 0x7f2f0806fad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35664/0 (socket says 172.21.15.45:35664) 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f077fe640 1 -- 172.21.15.45:0/3181047257 learned_addr learned my addr 172.21.15.45:0/3181047257 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f07fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2f0813c010 0x7f2f081d4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33770/0 (socket says 172.21.15.45:33770) 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 client.?.objecter ms_handle_connect 0x7f2f0813c010 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 client.?.objecter ms_handle_connect 0x7f2f0809fc40 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946584405 0 0) 0x7f2f08134e70 con 0x7f2f0809fc40 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): my global_id is 28502 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ef4004ec0 con 0x7f2f0809fc40 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 client.?.objecter ms_handle_connect 0x7f2f08135950 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3317050927 0 0) 0x7f2f080708f0 con 0x7f2f0813c010 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): my global_id is 94206 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f08134e70 con 0x7f2f0813c010 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1819876054 0 0) 0x7f2ef4004ec0 con 0x7f2f0809fc40 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f080708f0 con 0x7f2f0809fc40 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2efc001320 con 0x7f2f0809fc40 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): dump: 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:15.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3564593759 0 0) 0x7f2f0813b8d0 con 0x7f2f08135950 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): my global_id is 22252 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ef4004ec0 con 0x7f2f08135950 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2089379383 0 0) 0x7f2f08134e70 con 0x7f2f0813c010 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f0813b8d0 con 0x7f2f0813c010 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f00001ef0 con 0x7f2f0813c010 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient(hunting): dump: 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3135187924 0 0) 0x7f2f080708f0 con 0x7f2f0809fc40 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 >> v1:172.21.15.46:6789/0 conn(0x7f2f08135950 legacy=0x7f2f080701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 >> v1:172.21.15.45:6789/0 conn(0x7f2f0813c010 legacy=0x7f2f081d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient: _finish_hunting 0 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 monclient: found mon.c 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:15.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f080738f0 con 0x7f2f0809fc40 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient: _finish_auth 0 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient: _check_auth_tickets 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 5 monclient: authenticate success, global_id 28502 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient: _renew_subs 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f080720f0 con 0x7f2f0809fc40 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 client.28502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient: _renew_subs 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f081d62c0 con 0x7f2f0809fc40 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 librados: init done 2024-09-16T03:07:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2efc0047f0 con 0x7f2f0809fc40 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2efc004b40 con 0x7f2f0809fc40 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 monclient: dump: 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f05ffb640 4 set_mon_vals no callback set 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ee80065d0 con 0x7f2ee8002870 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 client.28502.objecter ms_handle_connect 0x7f2ee8002870 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 10 client.28502.objecter resend_mon_ops 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.008+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f2efc010bb0 con 0x7f2f0809fc40 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(88..88 src has 1..88) ==== 7967+0+0 (unknown 1418569008 0 0) 0x7f2efc04c100 con 0x7f2f0809fc40 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 client.28502.objecter ms_dispatch 0x7f2f08000ba0 osd_map(88..88 src has 1..88) 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 3 client.28502.objecter handle_osd_map got epochs [88,88] > 0 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 3 client.28502.objecter handle_osd_map decoding full epoch 88 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 20 client.28502.objecter dump_active .. 0 homeless 2024-09-16T03:07:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901956883 0 0) 0x7f2ee80065d0 con 0x7f2ee8002870 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 monclient(hunting): my global_id is 28505 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f080708f0 con 0x7f2ee8002870 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2f07fff640 10 monclient: get_auth_request con 0x7f2ef403e2b0 auth_method 0 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1009218773 0 0) 0x7f2f080708f0 con 0x7f2ee8002870 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ee80065d0 con 0x7f2ee8002870 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2ef8001780 con 0x7f2ee8002870 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 monclient: dump: 2024-09-16T03:07:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 513935283 0 0) 0x7f2ee80065d0 con 0x7f2ee8002870 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ee800a170 con 0x7f2ee8002870 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 client.28502.objecter ms_handle_connect 0x7f2ef403e2b0 2024-09-16T03:07:15.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2ef80015e0 con 0x7f2ee8002870 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 >> v1:172.21.15.45:6790/0 conn(0x7f2ee8002870 legacy=0x7f2ee8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.012+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 >> v1:172.21.15.45:6790/0 conn(0x7f2ee8002870 legacy=0x7f2ee8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2f0dfc9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ee8009e60 con 0x7f2ee8009120 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 10 client.28502.objecter ms_handle_connect 0x7f2ee8009120 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 10 client.28502.objecter resend_mon_ops 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2441571576 0 0) 0x7f2ee8009e60 con 0x7f2ee8009120 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 10 monclient(hunting): my global_id is 28508 2024-09-16T03:07:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ee80065d0 con 0x7f2ee8009120 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2941604062 0 0) 0x7f2ee80065d0 con 0x7f2ee8009120 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ee8009e60 con 0x7f2ee8009120 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2ef8003990 con 0x7f2ee8009120 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 10 monclient: dump: 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.036 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4203879450 0 0) 0x7f2ee8009e60 con 0x7f2ee8009120 2024-09-16T03:07:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.016+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ee800c400 con 0x7f2ee8009120 2024-09-16T03:07:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 4090268785) 0x7f2ef8005bd0 con 0x7f2ee8009120 2024-09-16T03:07:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2eeffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:07:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2eeffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 >> v1:172.21.15.45:6790/0 conn(0x7f2ee8009120 legacy=0x7f2ee80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2eeffff640 1 -- 172.21.15.45:0/3181047257 >> v1:172.21.15.45:6790/0 conn(0x7f2ee8009120 legacy=0x7f2ee80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 12, 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 156872, 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.037 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 3, 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:15.038 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:15.039 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:07:15.040 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:15.041 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:15.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.043 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:07:15.044 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.045 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:15.046 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:15.047 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 8 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:07:15.048 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:07:15.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 10 librados: watch_flush enter 2024-09-16T03:07:15.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 10 librados: watch_flush exit 2024-09-16T03:07:15.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 20 client.28502.objecter shutdown clearing up homeless session... 2024-09-16T03:07:15.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 10 client.28502.objecter successfully canceled tick 2024-09-16T03:07:15.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 >> v1:172.21.15.46:6812/149360280 conn(0x7f2ef403e2b0 legacy=0x7f2ef4040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 10 monclient: shutdown 2024-09-16T03:07:15.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 >> v1:172.21.15.45:6790/0 conn(0x7f2f0809fc40 legacy=0x7f2f0806fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f07fff640 1 -- 172.21.15.45:0/3181047257 reap_dead start 2024-09-16T03:07:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f07fff640 1 -- 172.21.15.45:0/3181047257 reap_dead start 2024-09-16T03:07:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 shutdown_connections 2024-09-16T03:07:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 >> 172.21.15.45:0/3181047257 conn(0x7f2f080a2bd0 msgr2=0x7f2f0813ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 shutdown_connections 2024-09-16T03:07:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 1 -- 172.21.15.45:0/3181047257 wait complete. 2024-09-16T03:07:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.020+0000 7f2f0dfc9640 1 librados: shutdown 2024-09-16T03:07:15.051 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:07:15.051 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:07:15.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:07:15.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient: get_monmap_and_config 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient: build_initial_monmap 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient: monmap: 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:15.186490+0000 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:15.186490+0000 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:15.188 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 1 Processor -- start 2024-09-16T03:07:15.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 1 -- start start 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient: init 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient(hunting): picked mon.noname-a con 0x7f22d4137c90 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient(hunting): picked mon.noname-c con 0x7f22d4074bc0 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient(hunting): picked mon.noname-b con 0x7f22d413e310 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d409da80 con 0x7f22d4137c90 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d40a5c40 con 0x7f22d4074bc0 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d409dcc0 con 0x7f22d413e310 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22dad89640 10 monclient(hunting): authenticate will time out at 1125.835615s 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22d8b25640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f22d4137c90 0x7f22d4138070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33786/0 (socket says 172.21.15.45:33786) 2024-09-16T03:07:15.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.184+0000 7f22d8b25640 1 -- 172.21.15.45:0/606987306 learned_addr learned my addr 172.21.15.45:0/606987306 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3266655473 0 0) 0x7f22d409dcc0 con 0x7f22d413e310 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): my global_id is 28511 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c0003530 con 0x7f22d413e310 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2631917335 0 0) 0x7f22d409da80 con 0x7f22d4137c90 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): my global_id is 94209 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d409dcc0 con 0x7f22d4137c90 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418843567 0 0) 0x7f22d40a5c40 con 0x7f22d4074bc0 2024-09-16T03:07:15.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient(hunting): my global_id is 22255 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d409da80 con 0x7f22d4074bc0 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 827057804 0 0) 0x7f22c0003530 con 0x7f22d413e310 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 >> v1:172.21.15.46:6789/0 conn(0x7f22d4074bc0 legacy=0x7f22d413cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 >> v1:172.21.15.45:6789/0 conn(0x7f22d4137c90 legacy=0x7f22d4138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 monclient: found mon.noname-b 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d4073260 con 0x7f22d413e310 2024-09-16T03:07:15.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: _finish_auth 0 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 5 monclient: authenticate success, global_id 28511 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22cc001950 con 0x7f22d413e310 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: dump: 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.200 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22cc001b70 con 0x7f22d413e310 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 1 -- 172.21.15.45:0/606987306 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22cc001f40 con 0x7f22d413e310 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 4 set_mon_vals no callback set 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d37fe640 10 monclient: dump: 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: shutdown 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- 172.21.15.45:0/606987306 >> v1:172.21.15.45:6790/0 conn(0x7f22d413e310 legacy=0x7f22d4140700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- 172.21.15.45:0/606987306 shutdown_connections 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- 172.21.15.45:0/606987306 >> 172.21.15.45:0/606987306 conn(0x7f22d40aa510 msgr2=0x7f22d40aa920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- 172.21.15.45:0/606987306 shutdown_connections 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- 172.21.15.45:0/606987306 wait complete. 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: build_initial_monmap 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: monmap: 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 librados: starting msgr at 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 librados: starting objecter 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 Processor -- start 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- start start 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 librados: setting wanted keys 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 librados: calling monclient init 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: init 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient(hunting): picked mon.noname-c con 0x7f22d4074bc0 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:15.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient(hunting): picked mon.noname-a con 0x7f22d4137c90 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient(hunting): picked mon.noname-b con 0x7f22d413e310 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d413dca0 con 0x7f22d4137c90 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d413be40 con 0x7f22d413e310 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d41397e0 con 0x7f22d4074bc0 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient(hunting): authenticate will time out at 1125.838795s 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d8b25640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f22d4074bc0 0x7f22d41389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35712/0 (socket says 172.21.15.45:35712) 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d3fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f22d4137c90 0x7f22d41390d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33790/0 (socket says 172.21.15.45:33790) 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d8b25640 1 -- 172.21.15.45:0/730038606 learned_addr learned my addr 172.21.15.45:0/730038606 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 client.?.objecter ms_handle_connect 0x7f22d4074bc0 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 client.?.objecter ms_handle_connect 0x7f22d4137c90 2024-09-16T03:07:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 client.?.objecter ms_handle_connect 0x7f22d413e310 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250008551 0 0) 0x7f22d409da80 con 0x7f22d4137c90 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): my global_id is 94212 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22b4004ef0 con 0x7f22d4137c90 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 612166307 0 0) 0x7f22d41397e0 con 0x7f22d4074bc0 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): my global_id is 28514 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d409da80 con 0x7f22d4074bc0 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899159891 0 0) 0x7f22d413be40 con 0x7f22d413e310 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): my global_id is 22258 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d41397e0 con 0x7f22d413e310 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2708728474 0 0) 0x7f22b4004ef0 con 0x7f22d4137c90 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22d413be40 con 0x7f22d4137c90 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22c4001980 con 0x7f22d4137c90 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): dump: 2024-09-16T03:07:15.207 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2622823580 0 0) 0x7f22d409da80 con 0x7f22d4074bc0 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22b4004ef0 con 0x7f22d4074bc0 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22bc001980 con 0x7f22d4074bc0 2024-09-16T03:07:15.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): dump: 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1668243367 0 0) 0x7f22d41397e0 con 0x7f22d413e310 2024-09-16T03:07:15.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22d409da80 con 0x7f22d413e310 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22cc001d20 con 0x7f22d413e310 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient(hunting): dump: 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1107867748 0 0) 0x7f22d413be40 con 0x7f22d4137c90 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 >> v1:172.21.15.45:6790/0 conn(0x7f22d4074bc0 legacy=0x7f22d41389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 >> v1:172.21.15.46:6789/0 conn(0x7f22d413e310 legacy=0x7f22d41d4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient: _finish_hunting 0 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 monclient: found mon.a 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d4139b00 con 0x7f22d4137c90 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient: _finish_auth 0 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 5 monclient: authenticate success, global_id 94212 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: _renew_subs 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:15.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22d4141b60 con 0x7f22d4137c90 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 client.94212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: _renew_subs 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f22d413a8a0 con 0x7f22d4137c90 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22dad89640 1 librados: init done 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22c4001c80 con 0x7f22d4137c90 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.188+0000 7f22d17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22c4003a70 con 0x7f22d4137c90 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d2ffd640 4 set_mon_vals no callback set 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient: dump: 2024-09-16T03:07:15.212 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22dad89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:15.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22b80065d0 con 0x7f22b8002870 2024-09-16T03:07:15.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f22c4003c90 con 0x7f22d4137c90 2024-09-16T03:07:15.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 client.94212.objecter ms_handle_connect 0x7f22b8002870 2024-09-16T03:07:15.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 client.94212.objecter resend_mon_ops 2024-09-16T03:07:15.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(89..89 src has 1..89) ==== 8078+0+0 (unknown 930600271 0 0) 0x7f22c404c040 con 0x7f22d4137c90 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 client.94212.objecter ms_dispatch 0x7f22d4000ba0 osd_map(89..89 src has 1..89) 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 3 client.94212.objecter handle_osd_map got epochs [89,89] > 0 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 3 client.94212.objecter handle_osd_map decoding full epoch 89 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 20 client.94212.objecter dump_active .. 0 homeless 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 476603707 0 0) 0x7f22b80065d0 con 0x7f22b8002870 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient(hunting): my global_id is 22261 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d413be40 con 0x7f22b8002870 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d9326640 10 monclient: get_auth_request con 0x7f22b403e320 auth_method 0 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1758582314 0 0) 0x7f22d413be40 con 0x7f22b8002870 2024-09-16T03:07:15.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22b80065d0 con 0x7f22b8002870 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22bc004990 con 0x7f22b8002870 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient: dump: 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.218 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 client.94212.objecter ms_handle_connect 0x7f22b403e320 2024-09-16T03:07:15.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 819986118 0 0) 0x7f22b80065d0 con 0x7f22b8002870 2024-09-16T03:07:15.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22b800a170 con 0x7f22b8002870 2024-09-16T03:07:15.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f22bc006030 con 0x7f22b8002870 2024-09-16T03:07:15.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:15.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:15.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 >> v1:172.21.15.46:6789/0 conn(0x7f22b8002870 legacy=0x7f22b8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.192+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 >> v1:172.21.15.46:6789/0 conn(0x7f22b8002870 legacy=0x7f22b8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22b8009e60 con 0x7f22b8009120 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 client.94212.objecter ms_handle_connect 0x7f22b8009120 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 client.94212.objecter resend_mon_ops 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2699726485 0 0) 0x7f22b8009e60 con 0x7f22b8009120 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 monclient(hunting): my global_id is 22264 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22b80065d0 con 0x7f22b8009120 2024-09-16T03:07:15.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3190636441 0 0) 0x7f22b80065d0 con 0x7f22b8009120 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22b8009e60 con 0x7f22b8009120 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22bc006840 con 0x7f22b8009120 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 monclient: dump: 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.222 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1058119276 0 0) 0x7f22b8009e60 con 0x7f22b8009120 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22b800c3b0 con 0x7f22b8009120 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 3977209422) 0x7f22bc006790 con 0x7f22b8009120 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 >> v1:172.21.15.46:6789/0 conn(0x7f22b8009120 legacy=0x7f22b80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d17fa640 1 -- 172.21.15.45:0/730038606 >> v1:172.21.15.46:6789/0 conn(0x7f22b8009120 legacy=0x7f22b80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 12, 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 157005, 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 3, 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:07:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:07:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:07:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:07:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:07:15.235 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:07:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 10 librados: watch_flush enter 2024-09-16T03:07:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 10 librados: watch_flush exit 2024-09-16T03:07:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 20 client.94212.objecter shutdown clearing up homeless session... 2024-09-16T03:07:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 10 client.94212.objecter successfully canceled tick 2024-09-16T03:07:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 >> v1:172.21.15.46:6812/149360280 conn(0x7f22b403e320 legacy=0x7f22b40407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 10 monclient: shutdown 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 >> v1:172.21.15.45:6789/0 conn(0x7f22d4137c90 legacy=0x7f22d41390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d9326640 1 -- 172.21.15.45:0/730038606 reap_dead start 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22d9326640 1 -- 172.21.15.45:0/730038606 reap_dead start 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 shutdown_connections 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 >> 172.21.15.45:0/730038606 conn(0x7f22d40aa510 msgr2=0x7f22d40a0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 shutdown_connections 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 1 -- 172.21.15.45:0/730038606 wait complete. 2024-09-16T03:07:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.200+0000 7f22dad89640 1 librados: shutdown 2024-09-16T03:07:15.237 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:07:15.238 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T03:07:15.238 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:07:15.238 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T03:07:15.238 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-16T03:07:15.238 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-16T03:07:15.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient: get_monmap_and_config 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient: build_initial_monmap 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient: monmap: 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:15.354529+0000 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:15.354529+0000 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:15.356 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 1 Processor -- start 2024-09-16T03:07:15.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 1 -- start start 2024-09-16T03:07:15.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient: init 2024-09-16T03:07:15.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:15.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient(hunting): picked mon.noname-a con 0x7f533009c570 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient(hunting): picked mon.noname-b con 0x7f5330134320 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient(hunting): picked mon.noname-c con 0x7f53301347d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5330135150 con 0x7f533009c570 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5330074470 con 0x7f53301347d0 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5330074750 con 0x7f5330134320 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.352+0000 7f5338773640 10 monclient(hunting): authenticate will time out at 1126.003909s 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533650f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f533009c570 0x7f533009c950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33804/0 (socket says 172.21.15.45:33804) 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533650f640 1 -- 172.21.15.45:0/2542348369 learned_addr learned my addr 172.21.15.45:0/2542348369 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 75689193 0 0) 0x7f5330074750 con 0x7f5330134320 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): my global_id is 28517 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f531c003530 con 0x7f5330134320 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2389019659 0 0) 0x7f532c001120 con 0x7f533009c570 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): my global_id is 94215 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5330074750 con 0x7f533009c570 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 219992247 0 0) 0x7f5330074470 con 0x7f53301347d0 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient(hunting): my global_id is 22267 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f532c001120 con 0x7f53301347d0 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3551965050 0 0) 0x7f5330074750 con 0x7f533009c570 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 >> v1:172.21.15.45:6790/0 conn(0x7f5330134320 legacy=0x7f533013cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 >> v1:172.21.15.46:6789/0 conn(0x7f53301347d0 legacy=0x7f53301402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: _finish_hunting 0 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 monclient: found mon.noname-a 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5330074a30 con 0x7f533009c570 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: _finish_auth 0 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: _check_auth_tickets 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f532c0017d0 con 0x7f533009c570 2024-09-16T03:07:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 5 monclient: authenticate success, global_id 94215 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: dump: 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f532c001970 con 0x7f533009c570 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 1 -- 172.21.15.45:0/2542348369 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f532c0048b0 con 0x7f533009c570 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f533550d640 10 monclient: dump: 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.369 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 4 set_mon_vals no callback set 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient: shutdown 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 -- 172.21.15.45:0/2542348369 >> v1:172.21.15.45:6789/0 conn(0x7f533009c570 legacy=0x7f533009c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 -- 172.21.15.45:0/2542348369 shutdown_connections 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 -- 172.21.15.45:0/2542348369 >> 172.21.15.45:0/2542348369 conn(0x7f53300a2bd0 msgr2=0x7f53300a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 -- 172.21.15.45:0/2542348369 shutdown_connections 2024-09-16T03:07:15.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 -- 172.21.15.45:0/2542348369 wait complete. 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient: build_initial_monmap 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient: monmap: 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 librados: starting msgr at 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 librados: starting objecter 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 Processor -- start 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 -- start start 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 librados: setting wanted keys 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 librados: calling monclient init 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient: init 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient(hunting): picked mon.noname-b con 0x7f533009c570 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient(hunting): picked mon.noname-a con 0x7f5330134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient(hunting): picked mon.noname-c con 0x7f53301347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5330133410 con 0x7f5330134320 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53301321d0 con 0x7f533009c570 2024-09-16T03:07:15.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5330131d50 con 0x7f53301347d0 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5338773640 10 monclient(hunting): authenticate will time out at 1126.007286s 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5335d0e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5330134320 0x7f5330130590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33814/0 (socket says 172.21.15.45:33814) 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5335d0e640 1 -- 172.21.15.45:0/2102497262 learned_addr learned my addr 172.21.15.45:0/2102497262 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 client.?.objecter ms_handle_connect 0x7f5330134320 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 client.?.objecter ms_handle_connect 0x7f53301347d0 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3640961739 0 0) 0x7f5330133410 con 0x7f5330134320 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 monclient(hunting): my global_id is 94218 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5314004ec0 con 0x7f5330134320 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 client.?.objecter ms_handle_connect 0x7f533009c570 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455988689 0 0) 0x7f5330131d50 con 0x7f53301347d0 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 10 monclient(hunting): my global_id is 28520 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5330133410 con 0x7f53301347d0 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.356+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3114185151 0 0) 0x7f5314004ec0 con 0x7f5330134320 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5330131d50 con 0x7f5330134320 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5320001420 con 0x7f5330134320 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): dump: 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815242009 0 0) 0x7f53301321d0 con 0x7f533009c570 2024-09-16T03:07:15.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): my global_id is 22270 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5314004ec0 con 0x7f533009c570 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 379957369 0 0) 0x7f5330133410 con 0x7f53301347d0 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53301321d0 con 0x7f53301347d0 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5328001c00 con 0x7f53301347d0 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): dump: 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.376 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1616514537 0 0) 0x7f5330131d50 con 0x7f5330134320 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 >> v1:172.21.15.45:6790/0 conn(0x7f53301347d0 legacy=0x7f53301d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 >> v1:172.21.15.46:6789/0 conn(0x7f533009c570 legacy=0x7f533012fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 monclient: found mon.a 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53301336f0 con 0x7f5330134320 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: _finish_auth 0 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 5 monclient: authenticate success, global_id 94218 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 10 monclient: _renew_subs 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5330130ca0 con 0x7f5330134320 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 10 client.94218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 10 monclient: _renew_subs 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5330099940 con 0x7f5330134320 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 1 librados: init done 2024-09-16T03:07:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5320001680 con 0x7f5330134320 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5320003a50 con 0x7f5330134320 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: dump: 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5334d0c640 4 set_mon_vals no callback set 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f53180065d0 con 0x7f5318002870 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f5320003c70 con 0x7f5330134320 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 client.94218.objecter ms_handle_connect 0x7f5318002870 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 client.94218.objecter resend_mon_ops 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(89..89 src has 1..89) ==== 8078+0+0 (unknown 930600271 0 0) 0x7f532004c2c0 con 0x7f5330134320 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 client.94218.objecter ms_dispatch 0x7f5330000ba0 osd_map(89..89 src has 1..89) 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 3 client.94218.objecter handle_osd_map got epochs [89,89] > 0 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 3 client.94218.objecter handle_osd_map decoding full epoch 89 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 20 client.94218.objecter dump_active .. 0 homeless 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538130228 0 0) 0x7f53180065d0 con 0x7f5318002870 2024-09-16T03:07:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient(hunting): my global_id is 94221 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5330131d50 con 0x7f5318002870 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2746325218 0 0) 0x7f5330131d50 con 0x7f5318002870 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53180065d0 con 0x7f5318002870 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f532c001790 con 0x7f5318002870 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5336d10640 10 monclient: get_auth_request con 0x7f531403e260 auth_method 0 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: dump: 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 311443185 0 0) 0x7f53180065d0 con 0x7f5318002870 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f531800a170 con 0x7f5318002870 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 client.94218.objecter ms_handle_connect 0x7f531403e260 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f532c001a90 con 0x7f5318002870 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 >> v1:172.21.15.45:6789/0 conn(0x7f5318002870 legacy=0x7f5318004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.360+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 >> v1:172.21.15.45:6789/0 conn(0x7f5318002870 legacy=0x7f5318004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:15.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:07:15.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5318009ea0 con 0x7f5318009160 2024-09-16T03:07:15.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 client.94218.objecter ms_handle_connect 0x7f5318009160 2024-09-16T03:07:15.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 client.94218.objecter resend_mon_ops 2024-09-16T03:07:15.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3292537998 0 0) 0x7f5318009ea0 con 0x7f5318009160 2024-09-16T03:07:15.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 monclient(hunting): my global_id is 94224 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53180065d0 con 0x7f5318009160 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3444078733 0 0) 0x7f53180065d0 con 0x7f5318009160 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5318009ea0 con 0x7f5318009160 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f532c004d60 con 0x7f5318009160 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 monclient: dump: 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.385 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3860924522 0 0) 0x7f5318009ea0 con 0x7f5318009160 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f531800c430 con 0x7f5318009160 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (unknown 0 0 2946029456) 0x7f532c006060 con 0x7f5318009160 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 >> v1:172.21.15.45:6789/0 conn(0x7f5318009160 legacy=0x7f53180095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5326ffd640 1 -- 172.21.15.45:0/2102497262 >> v1:172.21.15.45:6789/0 conn(0x7f5318009160 legacy=0x7f53180095f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:15.387 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:07:15.387 INFO:teuthology.orchestra.run.smithi045.stdout: "ret": 0, 2024-09-16T03:07:15.387 INFO:teuthology.orchestra.run.smithi045.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T03:07:15.387 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:07:15.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 10 librados: watch_flush enter 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 10 librados: watch_flush exit 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 20 client.94218.objecter shutdown clearing up homeless session... 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 10 client.94218.objecter successfully canceled tick 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 >> v1:172.21.15.46:6812/149360280 conn(0x7f531403e260 legacy=0x7f5314040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 10 monclient: shutdown 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 >> v1:172.21.15.45:6789/0 conn(0x7f5330134320 legacy=0x7f5330130590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5336d10640 1 -- 172.21.15.45:0/2102497262 reap_dead start 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5336d10640 1 -- 172.21.15.45:0/2102497262 reap_dead start 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 shutdown_connections 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 >> 172.21.15.45:0/2102497262 conn(0x7f53300a2bd0 msgr2=0x7f533009f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 shutdown_connections 2024-09-16T03:07:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 1 -- 172.21.15.45:0/2102497262 wait complete. 2024-09-16T03:07:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.368+0000 7f5338773640 1 librados: shutdown 2024-09-16T03:07:15.390 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:07:15.390 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:07:15.390 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:07:15.401 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:07:15.401 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:07:15.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient: get_monmap_and_config 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient: build_initial_monmap 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient: monmap: 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:15.558539+0000 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:15.558539+0000 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:15.561 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 1 Processor -- start 2024-09-16T03:07:15.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 1 -- start start 2024-09-16T03:07:15.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient: init 2024-09-16T03:07:15.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:07:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient(hunting): picked mon.noname-c con 0x7f55cc09c570 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient(hunting): picked mon.noname-b con 0x7f55cc134320 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient(hunting): picked mon.noname-a con 0x7f55cc1347d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55cc135150 con 0x7f55cc1347d0 2024-09-16T03:07:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55cc074470 con 0x7f55cc09c570 2024-09-16T03:07:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55cc074750 con 0x7f55cc134320 2024-09-16T03:07:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.556+0000 7f55d0eaf640 10 monclient(hunting): authenticate will time out at 1126.207936s 2024-09-16T03:07:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55cad9d640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f55cc1347d0 legacy=0x7f55cc1402a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c9d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f55cc134320 0x7f55cc13cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35732/0 (socket says 172.21.15.45:35732) 2024-09-16T03:07:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c9d9b640 1 -- 172.21.15.45:0/3643809243 learned_addr learned my addr 172.21.15.45:0/3643809243 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2937732206 0 0) 0x7f55cc074750 con 0x7f55cc134320 2024-09-16T03:07:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient(hunting): my global_id is 28523 2024-09-16T03:07:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55b4003530 con 0x7f55cc134320 2024-09-16T03:07:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3844928237 0 0) 0x7f55cc074470 con 0x7f55cc09c570 2024-09-16T03:07:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient(hunting): my global_id is 22273 2024-09-16T03:07:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55cc074750 con 0x7f55cc09c570 2024-09-16T03:07:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 729568564 0 0) 0x7f55b4003530 con 0x7f55cc134320 2024-09-16T03:07:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 >> v1:172.21.15.46:6789/0 conn(0x7f55cc09c570 legacy=0x7f55cc09c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 >> v1:172.21.15.45:6789/0 conn(0x7f55cc1347d0 legacy=0x7f55cc1402a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:07:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: _finish_hunting 0 2024-09-16T03:07:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 monclient: found mon.noname-b 2024-09-16T03:07:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55cc074a30 con 0x7f55cc134320 2024-09-16T03:07:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: _finish_auth 0 2024-09-16T03:07:15.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: _check_auth_tickets 2024-09-16T03:07:15.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:15.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f55c00016a0 con 0x7f55cc134320 2024-09-16T03:07:15.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:07:15.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: dump: 2024-09-16T03:07:15.577 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.578 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.578 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.578 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.578 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.578 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 5 monclient: authenticate success, global_id 28523 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f55c00018c0 con 0x7f55cc134320 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 1 -- 172.21.15.45:0/3643809243 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f55c0001c90 con 0x7f55cc134320 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 4 set_mon_vals no callback set 2024-09-16T03:07:15.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c959a640 10 monclient: dump: 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient: shutdown 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- 172.21.15.45:0/3643809243 >> v1:172.21.15.45:6790/0 conn(0x7f55cc134320 legacy=0x7f55cc13cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- 172.21.15.45:0/3643809243 shutdown_connections 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- 172.21.15.45:0/3643809243 >> 172.21.15.45:0/3643809243 conn(0x7f55cc0a2bd0 msgr2=0x7f55cc0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- 172.21.15.45:0/3643809243 shutdown_connections 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- 172.21.15.45:0/3643809243 wait complete. 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient: build_initial_monmap 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient: monmap: 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 librados: starting msgr at 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 librados: starting objecter 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 Processor -- start 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- start start 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 librados: setting wanted keys 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 librados: calling monclient init 2024-09-16T03:07:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient: init 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient(hunting): picked mon.noname-c con 0x7f55cc09c570 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient(hunting): picked mon.noname-a con 0x7f55cc134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient(hunting): picked mon.noname-b con 0x7f55cc1347d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55cc099870 con 0x7f55cc134320 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55cc0a57d0 con 0x7f55cc1347d0 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55cc133fa0 con 0x7f55cc09c570 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient(hunting): authenticate will time out at 1126.211107s 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55c9d9b640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f55cc134320 legacy=0x7f55cc130840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55ca59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f55cc09c570 0x7f55cc130130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35746/0 (socket says 172.21.15.45:35746) 2024-09-16T03:07:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55ca59c640 1 -- 172.21.15.45:0/125690169 learned_addr learned my addr 172.21.15.45:0/125690169 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 client.?.objecter ms_handle_connect 0x7f55cc09c570 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 575776395 0 0) 0x7f55cc074750 con 0x7f55cc09c570 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): my global_id is 28526 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55ac004ec0 con 0x7f55cc09c570 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 client.?.objecter ms_handle_connect 0x7f55cc1347d0 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1774933004 0 0) 0x7f55ac004ec0 con 0x7f55cc09c570 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55cc074750 con 0x7f55cc09c570 2024-09-16T03:07:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f55bc001a20 con 0x7f55cc09c570 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): dump: 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.585 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4125335844 0 0) 0x7f55c4003d40 con 0x7f55cc1347d0 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient(hunting): my global_id is 22276 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55ac004ec0 con 0x7f55cc1347d0 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 832861130 0 0) 0x7f55cc074750 con 0x7f55cc09c570 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 >> v1:172.21.15.46:6789/0 conn(0x7f55cc1347d0 legacy=0x7f55cc1d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 >> v1:172.21.15.45:6789/0 conn(0x7f55cc134320 legacy=0x7f55cc130840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient: _finish_hunting 0 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 monclient: found mon.c 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55cc1338a0 con 0x7f55cc09c570 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient: _finish_auth 0 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 10 monclient: _check_auth_tickets 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 5 monclient: authenticate success, global_id 28526 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient: _renew_subs 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.560+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f55cc133b80 con 0x7f55cc09c570 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55d0eaf640 10 client.28526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55d0eaf640 10 monclient: _renew_subs 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55d0eaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f55cc1335c0 con 0x7f55cc09c570 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55d0eaf640 1 librados: init done 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f55bc0038d0 con 0x7f55cc09c570 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f55bc003ca0 con 0x7f55cc09c570 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55c8d99640 4 set_mon_vals no callback set 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 10 monclient: dump: 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55d0eaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55d0eaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55d0eaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b0005090 con 0x7f55cc09c570 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f55bc003ec0 con 0x7f55cc09c570 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(89..89 src has 1..89) ==== 8078+0+0 (unknown 930600271 0 0) 0x7f55bc04c330 con 0x7f55cc09c570 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 10 client.28526.objecter ms_dispatch 0x7f55cc000ba0 osd_map(89..89 src has 1..89) 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 3 client.28526.objecter handle_osd_map got epochs [89,89] > 0 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 3 client.28526.objecter handle_osd_map decoding full epoch 89 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 20 client.28526.objecter dump_active .. 0 homeless 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55c9d9b640 10 monclient: get_auth_request con 0x7f55ac03dfb0 auth_method 0 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.564+0000 7f55baffd640 10 client.28526.objecter ms_handle_connect 0x7f55ac03dfb0 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.568+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f55cc133fa0 con 0x7f55cc09c570 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.568+0000 7f55baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.568+0000 7f55baffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:15.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:07:15.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f55b0005380 con 0x7f55cc09c570 2024-09-16T03:07:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55baffd640 1 -- 172.21.15.45:0/125690169 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2425673917) 0x7f55bc016c70 con 0x7f55cc09c570 2024-09-16T03:07:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55baffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:15.808 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:07:15.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 10 librados: watch_flush enter 2024-09-16T03:07:15.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 10 librados: watch_flush exit 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 20 client.28526.objecter shutdown clearing up homeless session... 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 10 client.28526.objecter successfully canceled tick 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 >> v1:172.21.15.46:6812/149360280 conn(0x7f55ac03dfb0 legacy=0x7f55ac040470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 10 monclient: shutdown 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 >> v1:172.21.15.45:6790/0 conn(0x7f55cc09c570 legacy=0x7f55cc130130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 shutdown_connections 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.804+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 >> 172.21.15.45:0/125690169 conn(0x7f55cc0a2bd0 msgr2=0x7f55cc09f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.808+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 shutdown_connections 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.808+0000 7f55d0eaf640 1 -- 172.21.15.45:0/125690169 wait complete. 2024-09-16T03:07:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:15.808+0000 7f55d0eaf640 1 librados: shutdown 2024-09-16T03:07:18.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:07:18.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient: get_monmap_and_config 2024-09-16T03:07:18.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient: build_initial_monmap 2024-09-16T03:07:18.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient: monmap: 2024-09-16T03:07:18.959 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:18.959 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:18.959 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:18.957773+0000 2024-09-16T03:07:18.959 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:18.957773+0000 2024-09-16T03:07:18.959 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:18.960 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:18.960 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:18.960 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:18.960 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:18.960 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:18.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 1 Processor -- start 2024-09-16T03:07:18.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 1 -- start start 2024-09-16T03:07:18.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient: init 2024-09-16T03:07:18.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:18.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:07:18.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient(hunting): picked mon.noname-c con 0x7f9fd4135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:18.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient(hunting): picked mon.noname-a con 0x7f9fd409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:18.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient(hunting): picked mon.noname-b con 0x7f9fd413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:18.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd409f7e0 con 0x7f9fd409fc40 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd409b160 con 0x7f9fd4135950 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd409b440 con 0x7f9fd413c010 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fda3b7640 10 monclient(hunting): authenticate will time out at 1129.607215s 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd37fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f9fd409fc40 legacy=0x7f9fd413a8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd8954640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9fd413c010 0x7f9fd413e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35748/0 (socket says 172.21.15.45:35748) 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd8954640 1 -- 172.21.15.45:0/652555667 learned_addr learned my addr 172.21.15.45:0/652555667 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 1 -- 172.21.15.45:0/652555667 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799574290 0 0) 0x7f9fd409b440 con 0x7f9fd413c010 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient(hunting): my global_id is 28529 2024-09-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 1 -- 172.21.15.45:0/652555667 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fbc003530 con 0x7f9fd413c010 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 1 -- 172.21.15.45:0/652555667 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1296913409 0 0) 0x7f9fbc003530 con 0x7f9fd413c010 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 1 -- 172.21.15.45:0/652555667 >> v1:172.21.15.46:6789/0 conn(0x7f9fd4135950 legacy=0x7f9fd4135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 1 -- 172.21.15.45:0/652555667 >> v1:172.21.15.45:6789/0 conn(0x7f9fd409fc40 legacy=0x7f9fd413a8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 1 monclient: found mon.noname-b 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 1 -- 172.21.15.45:0/652555667 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd4136440 con 0x7f9fd413c010 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 1 -- 172.21.15.45:0/652555667 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9fcc0016a0 con 0x7f9fd413c010 2024-09-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.955+0000 7f9fd2ffd640 10 monclient: dump: 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 5 monclient: authenticate success, global_id 28529 2024-09-16T03:07:18.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd2ffd640 1 -- 172.21.15.45:0/652555667 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9fcc0018c0 con 0x7f9fd413c010 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd2ffd640 1 -- 172.21.15.45:0/652555667 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9fcc001d50 con 0x7f9fd413c010 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 4 set_mon_vals no callback set 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd2ffd640 10 monclient: dump: 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: shutdown 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- 172.21.15.45:0/652555667 >> v1:172.21.15.45:6790/0 conn(0x7f9fd413c010 legacy=0x7f9fd413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- 172.21.15.45:0/652555667 shutdown_connections 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- 172.21.15.45:0/652555667 >> 172.21.15.45:0/652555667 conn(0x7f9fd40a2bd0 msgr2=0x7f9fd40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- 172.21.15.45:0/652555667 shutdown_connections 2024-09-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- 172.21.15.45:0/652555667 wait complete. 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: build_initial_monmap 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: monmap: 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 librados: starting msgr at 2024-09-16T03:07:18.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 librados: starting objecter 2024-09-16T03:07:18.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 Processor -- start 2024-09-16T03:07:18.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- start start 2024-09-16T03:07:18.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 librados: setting wanted keys 2024-09-16T03:07:18.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 librados: calling monclient init 2024-09-16T03:07:18.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: init 2024-09-16T03:07:18.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:18.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:07:18.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient(hunting): picked mon.noname-a con 0x7f9fd409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:18.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient(hunting): picked mon.noname-c con 0x7f9fd4135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:18.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient(hunting): picked mon.noname-b con 0x7f9fd413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:18.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd40710e0 con 0x7f9fd409fc40 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd4071380 con 0x7f9fd413c010 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd3fff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f9fd409fc40 legacy=0x7f9fd41cad90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd4071660 con 0x7f9fd4135950 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient(hunting): authenticate will time out at 1129.610436s 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd37fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9fd4135950 0x7f9fd41cfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35758/0 (socket says 172.21.15.45:35758) 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd37fe640 1 -- 172.21.15.45:0/1087709321 learned_addr learned my addr 172.21.15.45:0/1087709321 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f9fd4135950 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881885654 0 0) 0x7f9fd4071660 con 0x7f9fd4135950 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:18.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): my global_id is 28532 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fc0004ec0 con 0x7f9fd4135950 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f9fd413c010 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1269852294 0 0) 0x7f9fc0004ec0 con 0x7f9fd4135950 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9fd4071660 con 0x7f9fd4135950 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9fc8004560 con 0x7f9fd4135950 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:07:18.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): dump: 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023352834 0 0) 0x7f9fd4071380 con 0x7f9fd413c010 2024-09-16T03:07:18.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient(hunting): my global_id is 22282 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fc0004ec0 con 0x7f9fd413c010 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2473640126 0 0) 0x7f9fd4071660 con 0x7f9fd4135950 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 >> v1:172.21.15.46:6789/0 conn(0x7f9fd413c010 legacy=0x7f9fd41d34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 >> v1:172.21.15.45:6789/0 conn(0x7f9fd409fc40 legacy=0x7f9fd41cad90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 monclient: found mon.c 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd4071940 con 0x7f9fd4135950 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient: _finish_auth 0 2024-09-16T03:07:18.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 5 monclient: authenticate success, global_id 28532 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: _renew_subs 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- 172.21.15.45:0/1087709321 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd4070420 con 0x7f9fd4135950 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 client.28532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: _renew_subs 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 -- 172.21.15.45:0/1087709321 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9fd406f3c0 con 0x7f9fd4135950 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fda3b7640 1 librados: init done 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9fc8004700 con 0x7f9fd4135950 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:18.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9fc8004b10 con 0x7f9fd4135950 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd0ff9640 10 monclient: dump: 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:18.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.959+0000 7f9fd27fc640 4 set_mon_vals no callback set 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fda3b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fda3b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fda3b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fda3b7640 1 -- 172.21.15.45:0/1087709321 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb4005090 con 0x7f9fd4135950 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f9fc8004d30 con 0x7f9fd4135950 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(89..89 src has 1..89) ==== 8078+0+0 (unknown 930600271 0 0) 0x7f9fc804c5d0 con 0x7f9fd4135950 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd0ff9640 10 client.28532.objecter ms_dispatch 0x7f9fd4000ba0 osd_map(89..89 src has 1..89) 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd0ff9640 3 client.28532.objecter handle_osd_map got epochs [89,89] > 0 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd0ff9640 3 client.28532.objecter handle_osd_map decoding full epoch 89 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd0ff9640 20 client.28532.objecter dump_active .. 0 homeless 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd3fff640 10 monclient: get_auth_request con 0x7f9fc003e0f0 auth_method 0 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd0ff9640 10 client.28532.objecter ms_handle_connect 0x7f9fc003e0f0 2024-09-16T03:07:18.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd3fff640 1 -- 172.21.15.45:0/1087709321 >> v1:172.21.15.46:6812/149360280 conn(0x7f9fc003e0f0 legacy=0x7f9fc00405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:07:18.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd3fff640 1 -- 172.21.15.45:0/1087709321 >> v1:172.21.15.46:6812/149360280 conn(0x7f9fc003e0f0 legacy=0x7f9fc00405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:07:18.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd3fff640 1 --1- 172.21.15.45:0/1087709321 >> v1:172.21.15.46:6812/149360280 conn(0x7f9fc003e0f0 0x7f9fc00405b0 :-1 s=OPENED pgs=3552 cs=1 l=1).handle_message read tag failed 2024-09-16T03:07:18.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd3fff640 1 --1- 172.21.15.45:0/1087709321 >> v1:172.21.15.46:6812/149360280 conn(0x7f9fc003e0f0 0x7f9fc00405b0 :-1 s=OPENED pgs=3552 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:07:18.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.963+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 >> v1:172.21.15.46:6812/149360280 conn(0x7f9fc003e0f0 legacy=0x7f9fc00405b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:18.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.967+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f9fc8016dd0 con 0x7f9fd4135950 2024-09-16T03:07:18.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.967+0000 7f9fd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:18.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:18.967+0000 7f9fd0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:19.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.199+0000 7f9fda3b7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:07:19.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.199+0000 7f9fda3b7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.199+0000 7f9fda3b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.199+0000 7f9fda3b7640 1 -- 172.21.15.45:0/1087709321 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9fb4005380 con 0x7f9fd4135950 2024-09-16T03:07:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.199+0000 7f9fd0ff9640 1 -- 172.21.15.45:0/1087709321 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 1607282962) 0x7f9fc80166d0 con 0x7f9fd4135950 2024-09-16T03:07:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.199+0000 7f9fd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:19.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.199+0000 7f9fd0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:19.205 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 10 librados: watch_flush enter 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 10 librados: watch_flush exit 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 20 client.28532.objecter shutdown clearing up homeless session... 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 10 client.28532.objecter successfully canceled tick 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 10 monclient: shutdown 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 1 -- 172.21.15.45:0/1087709321 >> v1:172.21.15.45:6790/0 conn(0x7f9fd4135950 legacy=0x7f9fd41cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 1 -- 172.21.15.45:0/1087709321 shutdown_connections 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 1 -- 172.21.15.45:0/1087709321 >> 172.21.15.45:0/1087709321 conn(0x7f9fd40a2bd0 msgr2=0x7f9fd413ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 1 -- 172.21.15.45:0/1087709321 shutdown_connections 2024-09-16T03:07:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 1 -- 172.21.15.45:0/1087709321 wait complete. 2024-09-16T03:07:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:19.203+0000 7f9fda3b7640 1 librados: shutdown 2024-09-16T03:07:22.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:07:22.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient: get_monmap_and_config 2024-09-16T03:07:22.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient: build_initial_monmap 2024-09-16T03:07:22.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient: monmap: 2024-09-16T03:07:22.355 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:22.355 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:22.356 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:22.353968+0000 2024-09-16T03:07:22.356 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:22.353968+0000 2024-09-16T03:07:22.356 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:22.356 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:22.356 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:22.356 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:22.356 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:22.356 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:22.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 1 Processor -- start 2024-09-16T03:07:22.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 1 -- start start 2024-09-16T03:07:22.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient: init 2024-09-16T03:07:22.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:22.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:07:22.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient(hunting): picked mon.noname-b con 0x7f14ec09b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:22.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient(hunting): picked mon.noname-a con 0x7f14ec09b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:22.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient(hunting): picked mon.noname-c con 0x7f14ec13b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:22.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec09cc20 con 0x7f14ec09b250 2024-09-16T03:07:22.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec09d820 con 0x7f14ec13b000 2024-09-16T03:07:22.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec09b6a0 con 0x7f14ec09b8e0 2024-09-16T03:07:22.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:22.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f4424640 10 monclient(hunting): authenticate will time out at 1133.003311s 2024-09-16T03:07:22.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f19bf640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f14ec09b250 legacy=0x7f14ec1398e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:22.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f21c0640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f14ec09b8e0 0x7f14ec09bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50490/0 (socket says 172.21.15.45:50490) 2024-09-16T03:07:22.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.351+0000 7f14f21c0640 1 -- 172.21.15.45:0/542434354 learned_addr learned my addr 172.21.15.45:0/542434354 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:22.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.555+0000 7f14f19bf640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14ec09b250 legacy=0x7f14ec1398e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:22.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:22.959+0000 7f14f19bf640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14ec09b250 legacy=0x7f14ec1398e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:23.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:23.763+0000 7f14f19bf640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14ec09b250 legacy=0x7f14ec1398e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 10 monclient(hunting): tick 2024-09-16T03:07:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:07:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 1 monclient(hunting): continuing hunt 2024-09-16T03:07:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6790/0 conn(0x7f14ec09b8e0 legacy=0x7f14ec09bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.46:6789/0 conn(0x7f14ec13b000 legacy=0x7f14ec13d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14ec09b250 legacy=0x7f14ec1398e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 10 monclient(hunting): picked mon.noname-a con 0x7f14d0003700 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 10 monclient(hunting): picked mon.noname-b con 0x7f14ec09b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 10 monclient(hunting): picked mon.noname-c con 0x7f14ec13b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec0a21f0 con 0x7f14d0003700 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec09cc20 con 0x7f14ec13b000 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d0010400 con 0x7f14ec09b8e0 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14dffff640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.351+0000 7f14f21c0640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:25.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.555+0000 7f14f21c0640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:25.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:25.959+0000 7f14f21c0640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:26.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:26.763+0000 7f14f21c0640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 10 monclient(hunting): tick 2024-09-16T03:07:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:07:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 1 monclient(hunting): continuing hunt 2024-09-16T03:07:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:07:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6790/0 conn(0x7f14ec09b8e0 legacy=0x7f14d00093e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.46:6789/0 conn(0x7f14ec13b000 legacy=0x7f14d000cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:07:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:07:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 10 monclient(hunting): picked mon.noname-b con 0x7f14d000f430 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:28.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 10 monclient(hunting): picked mon.noname-c con 0x7f14ec13b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:28.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 10 monclient(hunting): picked mon.noname-a con 0x7f14d0003700 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:28.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d0011700 con 0x7f14d0003700 2024-09-16T03:07:28.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec0a21f0 con 0x7f14ec13b000 2024-09-16T03:07:28.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 1 -- 172.21.15.45:0/542434354 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d0010130 con 0x7f14d000f430 2024-09-16T03:07:28.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14dffff640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:28.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.355+0000 7f14f29c1640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.559+0000 7f14f29c1640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:28.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:28.963+0000 7f14f29c1640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:29.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:29.767+0000 7f14f29c1640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:30.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 -- 172.21.15.45:0/542434354 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1437127262 0 0) 0x7f14d0010130 con 0x7f14d000f430 2024-09-16T03:07:30.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient(hunting): my global_id is 28535 2024-09-16T03:07:30.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 -- 172.21.15.45:0/542434354 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14d8003530 con 0x7f14d000f430 2024-09-16T03:07:30.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 -- 172.21.15.45:0/542434354 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3631695541 0 0) 0x7f14d8003530 con 0x7f14d000f430 2024-09-16T03:07:30.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.46:6789/0 conn(0x7f14ec13b000 legacy=0x7f14d000cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6789/0 conn(0x7f14d0003700 legacy=0x7f14d0005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: _finish_hunting 0 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 monclient: found mon.noname-b 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 -- 172.21.15.45:0/542434354 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d0008480 con 0x7f14d000f430 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: _finish_auth 0 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: _check_auth_tickets 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 -- 172.21.15.45:0/542434354 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f14e00016a0 con 0x7f14d000f430 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f4424640 5 monclient: authenticate success, global_id 28535 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f4424640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:30.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: dump: 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f4424640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 -- 172.21.15.45:0/542434354 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14e0001840 con 0x7f14d000f430 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 1 -- 172.21.15.45:0/542434354 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f14e0001cd0 con 0x7f14d000f430 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f11be640 10 monclient: dump: 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.173 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.173 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.173 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f4424640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:30.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.163+0000 7f14f4424640 4 set_mon_vals no callback set 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 10 monclient: shutdown 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 -- 172.21.15.45:0/542434354 >> v1:172.21.15.45:6790/0 conn(0x7f14d000f430 legacy=0x7f14d000f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 -- 172.21.15.45:0/542434354 shutdown_connections 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 -- 172.21.15.45:0/542434354 >> 172.21.15.45:0/542434354 conn(0x7f14ec0a2bd0 msgr2=0x7f14ec0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 -- 172.21.15.45:0/542434354 shutdown_connections 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 -- 172.21.15.45:0/542434354 wait complete. 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 10 monclient: build_initial_monmap 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 10 monclient: monmap: 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:30.176 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 librados: starting msgr at 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 librados: starting objecter 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 Processor -- start 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 -- start start 2024-09-16T03:07:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 librados: setting wanted keys 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 1 librados: calling monclient init 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.167+0000 7f14f4424640 10 monclient: init 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 10 monclient(hunting): picked mon.noname-b con 0x7f14d0003700 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 10 monclient(hunting): picked mon.noname-a con 0x7f14d000f430 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 10 monclient(hunting): picked mon.noname-c con 0x7f14ec13b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec098b50 con 0x7f14d000f430 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec098e30 con 0x7f14d0003700 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f19bf640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f14d000f430 legacy=0x7f14ec06eac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec0a47c0 con 0x7f14ec13b000 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f4424640 10 monclient(hunting): authenticate will time out at 1140.820754s 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f29c1640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f14ec13b000 0x7f14ec06f1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50508/0 (socket says 172.21.15.45:50508) 2024-09-16T03:07:30.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14f29c1640 1 -- 172.21.15.45:0/713094621 learned_addr learned my addr 172.21.15.45:0/713094621 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:30.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14deffd640 10 client.?.objecter ms_handle_connect 0x7f14ec13b000 2024-09-16T03:07:30.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14deffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:30.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14deffd640 10 client.?.objecter ms_handle_connect 0x7f14d0003700 2024-09-16T03:07:30.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.171+0000 7f14deffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3206152689 0 0) 0x7f14ec098e30 con 0x7f14d0003700 2024-09-16T03:07:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): my global_id is 22288 2024-09-16T03:07:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14d00116a0 con 0x7f14d0003700 2024-09-16T03:07:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2132391150 0 0) 0x7f14ec0a47c0 con 0x7f14ec13b000 2024-09-16T03:07:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): my global_id is 28538 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14ec098e30 con 0x7f14ec13b000 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3615704879 0 0) 0x7f14ec098e30 con 0x7f14ec13b000 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14ec0a47c0 con 0x7f14ec13b000 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f14e4001cb0 con 0x7f14ec13b000 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): dump: 2024-09-16T03:07:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.175+0000 7f14deffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 910898231 0 0) 0x7f14ec0a47c0 con 0x7f14ec13b000 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 >> v1:172.21.15.46:6789/0 conn(0x7f14d0003700 legacy=0x7f14ec06e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 >> v1:172.21.15.45:6789/0 conn(0x7f14d000f430 legacy=0x7f14ec06eac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:07:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 monclient: _finish_hunting 0 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 1 monclient: found mon.c 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ec074200 con 0x7f14ec13b000 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 monclient: _finish_auth 0 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 monclient: _check_auth_tickets 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 5 monclient: authenticate success, global_id 28538 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 10 monclient: _renew_subs 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f14ec09d2c0 con 0x7f14ec13b000 2024-09-16T03:07:30.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14e4003980 con 0x7f14ec13b000 2024-09-16T03:07:30.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:30.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 10 client.28538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:30.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f14e4003d50 con 0x7f14ec13b000 2024-09-16T03:07:30.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 10 monclient: _renew_subs 2024-09-16T03:07:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f14ec1d6220 con 0x7f14ec13b000 2024-09-16T03:07:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 1 librados: init done 2024-09-16T03:07:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f09bd640 4 set_mon_vals no callback set 2024-09-16T03:07:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 monclient: dump: 2024-09-16T03:07:30.188 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.188 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.188 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.188 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.188 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.188 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.189 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.189 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.189 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.189 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14ec06c3e0 con 0x7f14ec13b000 2024-09-16T03:07:30.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f14e4003f70 con 0x7f14ec13b000 2024-09-16T03:07:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(89..89 src has 1..89) ==== 8078+0+0 (unknown 930600271 0 0) 0x7f14e404c470 con 0x7f14ec13b000 2024-09-16T03:07:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 client.28538.objecter ms_dispatch 0x7f14ec000ba0 osd_map(89..89 src has 1..89) 2024-09-16T03:07:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 3 client.28538.objecter handle_osd_map got epochs [89,89] > 0 2024-09-16T03:07:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 3 client.28538.objecter handle_osd_map decoding full epoch 89 2024-09-16T03:07:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 20 client.28538.objecter dump_active .. 0 homeless 2024-09-16T03:07:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14f21c0640 10 monclient: get_auth_request con 0x7f14d003ec30 auth_method 0 2024-09-16T03:07:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.179+0000 7f14deffd640 10 client.28538.objecter ms_handle_connect 0x7f14d003ec30 2024-09-16T03:07:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.183+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f14e40166e0 con 0x7f14ec13b000 2024-09-16T03:07:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.183+0000 7f14deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.183+0000 7f14deffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: Running main() from gmock_main.cc 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [----------] Global test environment set-up. 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4816 ms) 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5031 ms) 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3539 ms) 2024-09-16T03:07:30.258 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5047 ms) 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (6979 ms) 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25412 ms total) 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5028 ms) 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5947 ms) 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8022 ms) 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-16T03:07:30.259 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6009 ms) 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4004 ms) 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi046-29522-11. 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: Waiting for snaps to purge. 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (33953 ms) 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (62963 ms total) 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [----------] Global test environment tear-down 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (88375 ms total) 2024-09-16T03:07:30.260 INFO:tasks.workunit.client.0.smithi046.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-16T03:07:30.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:07:30.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:30.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f14ec076990 con 0x7f14ec13b000 2024-09-16T03:07:30.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14deffd640 1 -- 172.21.15.45:0/713094621 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1113324263) 0x7f14e401ff30 con 0x7f14ec13b000 2024-09-16T03:07:30.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:07:30.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14deffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:30.474 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:07:30.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 10 librados: watch_flush enter 2024-09-16T03:07:30.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 10 librados: watch_flush exit 2024-09-16T03:07:30.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 20 client.28538.objecter shutdown clearing up homeless session... 2024-09-16T03:07:30.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 10 client.28538.objecter successfully canceled tick 2024-09-16T03:07:30.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 >> v1:172.21.15.46:6812/149360280 conn(0x7f14d003ec30 legacy=0x7f14d00410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 10 monclient: shutdown 2024-09-16T03:07:30.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:30.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 >> v1:172.21.15.45:6790/0 conn(0x7f14ec13b000 legacy=0x7f14ec06f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 shutdown_connections 2024-09-16T03:07:30.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 >> 172.21.15.45:0/713094621 conn(0x7f14ec0a2bd0 msgr2=0x7f14ec072980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:30.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 shutdown_connections 2024-09-16T03:07:30.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 1 -- 172.21.15.45:0/713094621 wait complete. 2024-09-16T03:07:30.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.471+0000 7f14f4424640 1 librados: shutdown 2024-09-16T03:07:30.493 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:07:30.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:07:30.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:07:30.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient: get_monmap_and_config 2024-09-16T03:07:30.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient: build_initial_monmap 2024-09-16T03:07:30.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient: monmap: 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:30.609179+0000 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:30.609179+0000 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:30.611 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 1 Processor -- start 2024-09-16T03:07:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 1 -- start start 2024-09-16T03:07:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient: init 2024-09-16T03:07:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient(hunting): picked mon.noname-a con 0x7f208009c570 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient(hunting): picked mon.noname-c con 0x7f2080134320 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient(hunting): picked mon.noname-b con 0x7f20801347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080135150 con 0x7f208009c570 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080074470 con 0x7f2080134320 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080074750 con 0x7f20801347d0 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient(hunting): authenticate will time out at 1141.258507s 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207f7fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f208009c570 legacy=0x7f208009c950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207ffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f20801347d0 0x7f20801402a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36386/0 (socket says 172.21.15.45:36386) 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207ffff640 1 -- 172.21.15.45:0/495868684 learned_addr learned my addr 172.21.15.45:0/495868684 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 -- 172.21.15.45:0/495868684 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3036879014 0 0) 0x7f2080074750 con 0x7f20801347d0 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient(hunting): my global_id is 28553 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 -- 172.21.15.45:0/495868684 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2068003530 con 0x7f20801347d0 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 -- 172.21.15.45:0/495868684 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2895354399 0 0) 0x7f2068003530 con 0x7f20801347d0 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 -- 172.21.15.45:0/495868684 >> v1:172.21.15.46:6789/0 conn(0x7f2080134320 legacy=0x7f208013cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 -- 172.21.15.45:0/495868684 >> v1:172.21.15.45:6789/0 conn(0x7f208009c570 legacy=0x7f208009c950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 monclient: found mon.noname-b 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 -- 172.21.15.45:0/495868684 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2080074a30 con 0x7f20801347d0 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:07:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 -- 172.21.15.45:0/495868684 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2078001a20 con 0x7f20801347d0 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 5 monclient: authenticate success, global_id 28553 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: dump: 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 -- 172.21.15.45:0/495868684 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2078001c40 con 0x7f20801347d0 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 1 -- 172.21.15.45:0/495868684 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2078002050 con 0x7f20801347d0 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f207e7fc640 10 monclient: dump: 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.607+0000 7f20861b1640 4 set_mon_vals no callback set 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: shutdown 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- 172.21.15.45:0/495868684 >> v1:172.21.15.45:6790/0 conn(0x7f20801347d0 legacy=0x7f20801402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- 172.21.15.45:0/495868684 shutdown_connections 2024-09-16T03:07:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- 172.21.15.45:0/495868684 >> 172.21.15.45:0/495868684 conn(0x7f20800a2bd0 msgr2=0x7f20800a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- 172.21.15.45:0/495868684 shutdown_connections 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- 172.21.15.45:0/495868684 wait complete. 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: build_initial_monmap 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: monmap: 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:30.621 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 librados: starting msgr at 2024-09-16T03:07:30.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 librados: starting objecter 2024-09-16T03:07:30.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 Processor -- start 2024-09-16T03:07:30.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- start start 2024-09-16T03:07:30.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 librados: setting wanted keys 2024-09-16T03:07:30.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 librados: calling monclient init 2024-09-16T03:07:30.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: init 2024-09-16T03:07:30.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:30.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:07:30.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient(hunting): picked mon.noname-a con 0x7f208009c570 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:30.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient(hunting): picked mon.noname-b con 0x7f2080134320 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:30.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient(hunting): picked mon.noname-c con 0x7f20801347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080099700 con 0x7f208009c570 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080133870 con 0x7f2080134320 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080130c90 con 0x7f20801347d0 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f207f7fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f208009c570 legacy=0x7f208012fe70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient(hunting): authenticate will time out at 1141.261717s 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f207ffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f20801347d0 0x7f20801d4e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36396/0 (socket says 172.21.15.45:36396) 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f207ffff640 1 -- 172.21.15.45:0/4213836461 learned_addr learned my addr 172.21.15.45:0/4213836461 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 client.?.objecter ms_handle_connect 0x7f20801347d0 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009720966 0 0) 0x7f2080130c90 con 0x7f20801347d0 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient(hunting): my global_id is 28556 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f206c004ec0 con 0x7f20801347d0 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 client.?.objecter ms_handle_connect 0x7f2080134320 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1565178464 0 0) 0x7f206c004ec0 con 0x7f20801347d0 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2080130c90 con 0x7f20801347d0 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2078001f90 con 0x7f20801347d0 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient(hunting): dump: 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.628 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 615400675 0 0) 0x7f2080130c90 con 0x7f20801347d0 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 >> v1:172.21.15.46:6789/0 conn(0x7f2080134320 legacy=0x7f2080130580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 >> v1:172.21.15.45:6789/0 conn(0x7f208009c570 legacy=0x7f208012fe70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient: _finish_hunting 0 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 monclient: found mon.c 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20800a57f0 con 0x7f20801347d0 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient: _finish_auth 0 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient: _check_auth_tickets 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 5 monclient: authenticate success, global_id 28556 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: _renew_subs 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2080133590 con 0x7f20801347d0 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 client.28556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: _renew_subs 2024-09-16T03:07:30.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2080132ec0 con 0x7f20801347d0 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 librados: init done 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2078001ba0 con 0x7f20801347d0 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2078004fc0 con 0x7f20801347d0 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f207dffb640 4 set_mon_vals no callback set 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 monclient: dump: 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.631 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f20600065d0 con 0x7f2060002870 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 client.28556.objecter ms_handle_connect 0x7f2060002870 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 client.28556.objecter resend_mon_ops 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f2078011030 con 0x7f20801347d0 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(90..90 src has 1..90) ==== 6985+0+0 (unknown 2649463755 0 0) 0x7f207804c030 con 0x7f20801347d0 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 10 client.28556.objecter ms_dispatch 0x7f2080000ba0 osd_map(90..90 src has 1..90) 2024-09-16T03:07:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 3 client.28556.objecter handle_osd_map got epochs [90,90] > 0 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.611+0000 7f2067fff640 3 client.28556.objecter handle_osd_map decoding full epoch 90 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 20 client.28556.objecter dump_active .. 0 homeless 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3791113045 0 0) 0x7f20600065d0 con 0x7f2060002870 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 monclient(hunting): my global_id is 28559 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2080130c90 con 0x7f2060002870 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f207effd640 10 monclient: get_auth_request con 0x7f206c03deb0 auth_method 0 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1092380280 0 0) 0x7f2080130c90 con 0x7f2060002870 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20600065d0 con 0x7f2060002870 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2070003fb0 con 0x7f2060002870 2024-09-16T03:07:30.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 monclient: dump: 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1447021125 0 0) 0x7f20600065d0 con 0x7f2060002870 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f206000a170 con 0x7f2060002870 2024-09-16T03:07:30.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 client.28556.objecter ms_handle_connect 0x7f206c03deb0 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f20700063d0 con 0x7f2060002870 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 >> v1:172.21.15.45:6790/0 conn(0x7f2060002870 legacy=0x7f2060004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.615+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 >> v1:172.21.15.45:6790/0 conn(0x7f2060002870 legacy=0x7f2060004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f20861b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2060009e60 con 0x7f2060009120 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 client.28556.objecter ms_handle_connect 0x7f2060009120 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 client.28556.objecter resend_mon_ops 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 784810088 0 0) 0x7f2060009e60 con 0x7f2060009120 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 monclient(hunting): my global_id is 28562 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20600065d0 con 0x7f2060009120 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 102105988 0 0) 0x7f20600065d0 con 0x7f2060009120 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2060009e60 con 0x7f2060009120 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f20700047a0 con 0x7f2060009120 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 monclient: dump: 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1185268636 0 0) 0x7f2060009e60 con 0x7f2060009120 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f206000c400 con 0x7f2060009120 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4291 (unknown 0 0 81327423) 0x7f2070005c50 con 0x7f2060009120 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 >> v1:172.21.15.45:6790/0 conn(0x7f2060009120 legacy=0x7f20600095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.619+0000 7f2067fff640 1 -- 172.21.15.45:0/4213836461 >> v1:172.21.15.45:6790/0 conn(0x7f2060009120 legacy=0x7f20600095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 16, 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 172472, 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:07:30.638 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.639 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.640 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:07:30.641 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:07:30.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:07:30.643 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.644 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.645 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:07:30.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.647 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 8 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:07:30.648 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 10 librados: watch_flush enter 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 10 librados: watch_flush exit 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 20 client.28556.objecter shutdown clearing up homeless session... 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 10 client.28556.objecter successfully canceled tick 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 >> v1:172.21.15.46:6812/149360280 conn(0x7f206c03deb0 legacy=0x7f206c040370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 10 monclient: shutdown 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 >> v1:172.21.15.45:6790/0 conn(0x7f20801347d0 legacy=0x7f20801d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f207ffff640 1 -- 172.21.15.45:0/4213836461 reap_dead start 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f207ffff640 1 -- 172.21.15.45:0/4213836461 reap_dead start 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 shutdown_connections 2024-09-16T03:07:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 >> 172.21.15.45:0/4213836461 conn(0x7f20800a2bd0 msgr2=0x7f208009f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:30.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 shutdown_connections 2024-09-16T03:07:30.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 1 -- 172.21.15.45:0/4213836461 wait complete. 2024-09-16T03:07:30.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.623+0000 7f20861b1640 1 librados: shutdown 2024-09-16T03:07:30.651 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:07:30.651 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:07:30.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:07:30.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.763+0000 7ff1f364a640 10 monclient: get_monmap_and_config 2024-09-16T03:07:30.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.763+0000 7ff1f364a640 10 monclient: build_initial_monmap 2024-09-16T03:07:30.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.763+0000 7ff1f364a640 10 monclient: monmap: 2024-09-16T03:07:30.769 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:30.769 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.769 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:07:30.768052+0000 2024-09-16T03:07:30.770 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:07:30.768052+0000 2024-09-16T03:07:30.770 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:30.770 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.770 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:30.770 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:07:30.770 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:07:30.770 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 Processor -- start 2024-09-16T03:07:30.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 -- start start 2024-09-16T03:07:30.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient: init 2024-09-16T03:07:30.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:30.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:07:30.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1ec09b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient(hunting): picked mon.noname-c con 0x7ff1ec09b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient(hunting): picked mon.noname-b con 0x7ff1ec13b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec09cc20 con 0x7ff1ec09b8e0 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec09d820 con 0x7ff1ec09b250 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec09b6a0 con 0x7ff1ec13b000 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f13e6640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7ff1ec09b8e0 legacy=0x7ff1ec09bcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient(hunting): authenticate will time out at 1141.417509s 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f1be7640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff1ec13b000 0x7ff1ec13d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36420/0 (socket says 172.21.15.45:36420) 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f1be7640 1 -- 172.21.15.45:0/1342123255 learned_addr learned my addr 172.21.15.45:0/1342123255 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 -- 172.21.15.45:0/1342123255 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311820798 0 0) 0x7ff1ec09b6a0 con 0x7ff1ec13b000 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient(hunting): my global_id is 28565 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 -- 172.21.15.45:0/1342123255 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1d8003530 con 0x7ff1ec13b000 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 -- 172.21.15.45:0/1342123255 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3682880577 0 0) 0x7ff1d8003530 con 0x7ff1ec13b000 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 -- 172.21.15.45:0/1342123255 >> v1:172.21.15.46:6789/0 conn(0x7ff1ec09b250 legacy=0x7ff1ec1398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 -- 172.21.15.45:0/1342123255 >> v1:172.21.15.45:6789/0 conn(0x7ff1ec09b8e0 legacy=0x7ff1ec09bcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 monclient: found mon.noname-b 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 -- 172.21.15.45:0/1342123255 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec0a21f0 con 0x7ff1ec13b000 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: _finish_auth 0 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:07:30.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 5 monclient: authenticate success, global_id 28565 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 -- 172.21.15.45:0/1342123255 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff1e4001a20 con 0x7ff1ec13b000 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: dump: 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 -- 172.21.15.45:0/1342123255 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1e4001c40 con 0x7ff1ec13b000 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 1 -- 172.21.15.45:0/1342123255 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff1e4002050 con 0x7ff1ec13b000 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient: get_monmap_and_config success 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 4 set_mon_vals no callback set 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1e3fff640 10 monclient: dump: 2024-09-16T03:07:30.777 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient: shutdown 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 -- 172.21.15.45:0/1342123255 >> v1:172.21.15.45:6790/0 conn(0x7ff1ec13b000 legacy=0x7ff1ec13d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 -- 172.21.15.45:0/1342123255 shutdown_connections 2024-09-16T03:07:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 -- 172.21.15.45:0/1342123255 >> 172.21.15.45:0/1342123255 conn(0x7ff1ec0a2bd0 msgr2=0x7ff1ec0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 -- 172.21.15.45:0/1342123255 shutdown_connections 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 -- 172.21.15.45:0/1342123255 wait complete. 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient: build_initial_monmap 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 10 monclient: monmap: 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:07:30.779 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:07:30.780 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 librados: starting msgr at 2024-09-16T03:07:30.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 librados: starting objecter 2024-09-16T03:07:30.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.767+0000 7ff1f364a640 1 Processor -- start 2024-09-16T03:07:30.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 -- start start 2024-09-16T03:07:30.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 librados: setting wanted keys 2024-09-16T03:07:30.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 librados: calling monclient init 2024-09-16T03:07:30.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient: init 2024-09-16T03:07:30.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient: _reopen_session rank -1 2024-09-16T03:07:30.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:07:30.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1ec09b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient(hunting): picked mon.noname-b con 0x7ff1ec09b8e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient(hunting): picked mon.noname-c con 0x7ff1ec13b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec09dd40 con 0x7ff1ec09b250 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec09e270 con 0x7ff1ec09b8e0 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f13e6640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7ff1ec09b250 legacy=0x7ff1ec1ccd60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec09d2c0 con 0x7ff1ec13b000 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient(hunting): _renew_subs 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient(hunting): authenticate will time out at 1141.420671s 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f1be7640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff1ec13b000 0x7ff1ec1d5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36430/0 (socket says 172.21.15.45:36430) 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f1be7640 1 -- 172.21.15.45:0/3514088362 learned_addr learned my addr 172.21.15.45:0/3514088362 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 client.?.objecter ms_handle_connect 0x7ff1ec13b000 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 928395035 0 0) 0x7ff1ec09d2c0 con 0x7ff1ec13b000 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): my global_id is 28568 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1d0004ec0 con 0x7ff1ec13b000 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 client.?.objecter ms_handle_connect 0x7ff1ec09b8e0 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 999260497 0 0) 0x7ff1d0004ec0 con 0x7ff1ec13b000 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1ec09d2c0 con 0x7ff1ec13b000 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff1e4001f90 con 0x7ff1ec13b000 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): dump: 2024-09-16T03:07:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.788 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.788 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.788 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.788 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068298733 0 0) 0x7ff1ec09e270 con 0x7ff1ec09b8e0 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): my global_id is 22309 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1d0004ec0 con 0x7ff1ec09b8e0 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4064714104 0 0) 0x7ff1ec09d2c0 con 0x7ff1ec13b000 2024-09-16T03:07:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 >> v1:172.21.15.46:6789/0 conn(0x7ff1ec09b8e0 legacy=0x7ff1ec1d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 >> v1:172.21.15.45:6789/0 conn(0x7ff1ec09b250 legacy=0x7ff1ec1ccd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 monclient: found mon.c 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec094c20 con 0x7ff1ec13b000 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 5 monclient: authenticate success, global_id 28568 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient: _renew_subs 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ec13d160 con 0x7ff1ec13b000 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 client.28568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient: _renew_subs 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1ec13c3f0 con 0x7ff1ec13b000 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 librados: init done 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1e4001ba0 con 0x7ff1ec13b000 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff1e4004f80 con 0x7ff1ec13b000 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: dump: 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e37fe640 4 set_mon_vals no callback set 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff1cc0065d0 con 0x7ff1cc002870 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7ff1e4010ff0 con 0x7ff1ec13b000 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 client.28568.objecter ms_handle_connect 0x7ff1cc002870 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 client.28568.objecter resend_mon_ops 2024-09-16T03:07:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(90..90 src has 1..90) ==== 6985+0+0 (unknown 2649463755 0 0) 0x7ff1e404c070 con 0x7ff1ec13b000 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 client.28568.objecter ms_dispatch 0x7ff1ec000ba0 osd_map(90..90 src has 1..90) 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 3 client.28568.objecter handle_osd_map got epochs [90,90] > 0 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 3 client.28568.objecter handle_osd_map decoding full epoch 90 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 20 client.28568.objecter dump_active .. 0 homeless 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 737087945 0 0) 0x7ff1cc0065d0 con 0x7ff1cc002870 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient(hunting): my global_id is 22312 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1ec09d2c0 con 0x7ff1cc002870 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1f0be5640 10 monclient: get_auth_request con 0x7ff1d003e090 auth_method 0 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3698011504 0 0) 0x7ff1ec09d2c0 con 0x7ff1cc002870 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1cc0065d0 con 0x7ff1cc002870 2024-09-16T03:07:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff1e8003ff0 con 0x7ff1cc002870 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 monclient: dump: 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.771+0000 7ff1e1ffb640 10 client.28568.objecter ms_handle_connect 0x7ff1d003e090 2024-09-16T03:07:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.775+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 373599871 0 0) 0x7ff1cc0065d0 con 0x7ff1cc002870 2024-09-16T03:07:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.775+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1cc00a170 con 0x7ff1cc002870 2024-09-16T03:07:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.775+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff1e8006450 con 0x7ff1cc002870 2024-09-16T03:07:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.775+0000 7ff1e1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:07:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.775+0000 7ff1e1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:07:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.775+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 >> v1:172.21.15.46:6789/0 conn(0x7ff1cc002870 legacy=0x7ff1cc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.775+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 >> v1:172.21.15.46:6789/0 conn(0x7ff1cc002870 legacy=0x7ff1cc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.779+0000 7ff1f364a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:07:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.779+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff1cc009e60 con 0x7ff1cc009120 2024-09-16T03:07:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 client.28568.objecter ms_handle_connect 0x7ff1cc009120 2024-09-16T03:07:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 client.28568.objecter resend_mon_ops 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3405657898 0 0) 0x7ff1cc009e60 con 0x7ff1cc009120 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 monclient(hunting): my global_id is 22315 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1cc0065d0 con 0x7ff1cc009120 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 184571473 0 0) 0x7ff1cc0065d0 con 0x7ff1cc009120 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1cc009e60 con 0x7ff1cc009120 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff1e8004820 con 0x7ff1cc009120 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 monclient: dump: 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3158275229 0 0) 0x7ff1cc009e60 con 0x7ff1cc009120 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1cc00c400 con 0x7ff1cc009120 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4126 (unknown 0 0 35771485) 0x7ff1e8005cd0 con 0x7ff1cc009120 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:07:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 >> v1:172.21.15.46:6789/0 conn(0x7ff1cc009120 legacy=0x7ff1cc0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1e1ffb640 1 -- 172.21.15.45:0/3514088362 >> v1:172.21.15.46:6789/0 conn(0x7ff1cc009120 legacy=0x7ff1cc0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 16, 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 172589, 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:07:30.799 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:30.800 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:30.801 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:07:30.802 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:07:30.803 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:30.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:07:30.806 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.807 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.808 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:07:30.808 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:07:30.808 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:07:30.808 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:07:30.808 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:07:30.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.808 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:30.809 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:07:30.810 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:07:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 10 librados: watch_flush enter 2024-09-16T03:07:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 10 librados: watch_flush exit 2024-09-16T03:07:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 20 client.28568.objecter shutdown clearing up homeless session... 2024-09-16T03:07:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 10 client.28568.objecter successfully canceled tick 2024-09-16T03:07:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 >> v1:172.21.15.46:6812/149360280 conn(0x7ff1d003e090 legacy=0x7ff1d0040550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 10 monclient: shutdown 2024-09-16T03:07:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:07:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 >> v1:172.21.15.45:6790/0 conn(0x7ff1ec13b000 legacy=0x7ff1ec1d5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:07:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f1be7640 1 -- 172.21.15.45:0/3514088362 reap_dead start 2024-09-16T03:07:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f1be7640 1 -- 172.21.15.45:0/3514088362 reap_dead start 2024-09-16T03:07:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 shutdown_connections 2024-09-16T03:07:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 >> 172.21.15.45:0/3514088362 conn(0x7ff1ec0a2bd0 msgr2=0x7ff1ec13de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:07:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 shutdown_connections 2024-09-16T03:07:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 1 -- 172.21.15.45:0/3514088362 wait complete. 2024-09-16T03:07:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:07:30.783+0000 7ff1f364a640 1 librados: shutdown 2024-09-16T03:07:30.813 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:07:30.813 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:07:31.264 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-16T03:07:31.265 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-16T03:07:31.265 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-16T03:07:31.265 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-16T03:07:31.265 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-16T03:07:31.265 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T03:07:31.265 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: 2024-09-16T03:07:31.265 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-16T03:07:31.265 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: seed 28644 2024-09-16T03:07:31.265 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (3 ms) 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (338 ms) 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (15 ms) 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (6 ms) 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (12 ms) 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-09-16T03:07:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (13 ms) 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (3 ms) 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (4 ms) 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-16T03:07:31.267 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (9 ms) 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2883 ms) 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (221 ms) 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: done waiting, doing copies 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: done waiting 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60577 ms) 2024-09-16T03:07:31.268 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (11 ms) 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (22482 ms) 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (1 ms) 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-16T03:07:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-16T03:07:32.249 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscP api_snapshots_pp: Running main() from gmock_main.cc 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: seed 28750 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2029 ms) 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2010 ms) 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2019 ms) 2024-09-16T03:07:32.250 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2000 ms) 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3548 ms) 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11606 ms total) 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4199 ms) 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3762 ms) 2024-09-16T03:07:32.251 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6027 ms) 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4014 ms) 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2027 ms) 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-16T03:07:32.252 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: deleting snap 15 in pool LibRadosSnapshotsSelfManagedPP_smithi046-28750-7 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17961 ms) 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (37990 ms total) 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (15 ms) 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4895 ms) 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4910 ms total) 2024-09-16T03:07:32.253 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: 2024-09-16T03:07:32.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-16T03:07:32.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-16T03:07:32.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3041 ms) 2024-09-16T03:07:32.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-16T03:07:32.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (16130 ms) 2024-09-16T03:07:32.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-16T03:07:32.254 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2008 ms) 2024-09-16T03:07:38.484 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshotsr()=9:89d3ae78:::11:head expected=9:89d3ae78:::11:head 2024-09-16T03:07:38.484 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-09-16T03:07:38.484 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=11 expected=11 2024-09-16T03:07:38.484 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:863748b0:::15:head 2024-09-16T03:07:38.484 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:863748b0:::15:head expected=9:863748b0:::15:head 2024-09-16T03:07:38.484 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-09-16T03:07:38.484 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=15 expected=15 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:6cac518f:::0:head expected=9:6cac518f:::0:head 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=0 expected=0 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:02547ec2:::1:head expected=9:02547ec2:::1:head 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=1 expected=1 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:62a1935d:::14:head expected=9:62a1935d:::14:head 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=14 expected=14 2024-09-16T03:07:38.485 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:5c6b0b28:::7:head expected=9:5c6b0b28:::7:head 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=7 expected=7 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:566253c9:::13:head 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:566253c9:::13:head expected=9:566253c9:::13:head 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=13 expected=13 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : cursor()=9:52ea6a34:::10:head expected=9:52ea6a34:::10:head 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: : entry=10 expected=10 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (480 ms) 2024-09-16T03:07:38.486 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (69633 ms) 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: split 0/5 -> MIN 9:33333333::::head 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: split 1/5 -> 9:33333333::::head 9:66666666::::head 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: split 2/5 -> 9:66666666::::head 9:99999999::::head 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: split 3/5 -> 9:99999999::::head 9:cccccccc::::head 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: split 4/5 -> 9:cccccccc::::head MAX 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (20205 ms) 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [----------] 7 tests from LibRadosList (90694 ms total) 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-16T03:07:38.487 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1078 ms) 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo1 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo2 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo3 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo4 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo5 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo1 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: myset foo6,foo7 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo7 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: foo6 2024-09-16T03:07:38.488 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns1:foo4 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns1:foo5 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns2:foo7 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns2:foo6 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: ns1:foo1 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: :foo1 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: :foo2 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: :foo3 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (56 ms) 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 1 0 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 10 0 2024-09-16T03:07:38.489 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 13 0 2024-09-16T03:07:44.330 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 7 0 2024-09-16T03:07:44.330 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 14 0 2024-09-16T03:07:44.330 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 0 0 2024-09-16T03:07:44.330 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 15 0 2024-09-16T03:07:44.330 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 11 0 2024-09-16T03:07:44.330 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 5 0 2024-09-16T03:07:44.330 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 8 0 2024-09-16T03:07:44.330 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 6 0 2024-09-16T03:07:44.330 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 3 0 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 4 0 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 12 0 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 9 0 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 2 0 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (87 ms) 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [----------] 3 tests from LibRadosListEC (1221 ms total) 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3013 ms) 2024-09-16T03:07:44.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [----------] 1 test from LibRadosListNP (3013 ms total) 2024-09-16T03:07:44.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: 2024-09-16T03:07:44.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [----------] Global test environment tear-down 2024-09-16T03:07:44.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [==========] 11 tests from 3 test suites ran. (102807 ms total) 2024-09-16T03:07:44.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_list: [ PASSED ] 11 tests. 2024-09-16T03:07:45.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: Running main() from gmock_main.cc 2024-09-16T03:07:45.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-16T03:07:45.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [----------] Global test environment set-up. 2024-09-16T03:07:45.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-16T03:07:45.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-16T03:07:45.338 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2218 ms) 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (2999 ms) 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3022 ms) 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3009 ms) 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2558 ms) 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3024 ms) 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3014 ms) 2024-09-16T03:07:45.339 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2933 ms) 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3042 ms) 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3036 ms) 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2945 ms) 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3060 ms) 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2944 ms) 2024-09-16T03:07:45.340 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3035 ms) 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.Flush (2979 ms) 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2982 ms) 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3007 ms) 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3012 ms) 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (8893 ms) 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T03:07:45.341 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (9913 ms) 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (17149 ms) 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3029 ms) 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3037 ms) 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2985 ms) 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3026 ms) 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3014 ms) 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [----------] 26 tests from LibRadosAio (103865 ms total) 2024-09-16T03:07:45.342 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: 2024-09-16T03:07:45.343 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-16T03:07:55.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-16T03:07:55.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2043 ms) 2024-09-16T03:07:55.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (23222 ms total) 2024-09-16T03:07:55.331 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4086 ms) 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3917 ms) 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4007 ms) 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12010 ms total) 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (113751 ms total) 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-16T03:07:55.332 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-16T03:07:55.333 INFO:tasks.workunit.client.0.smithi046.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T03:07:56.343 INFO:tasks.workunit.client.0.smithi046.stdout:P (86595 ms total) 2024-09-16T03:07:56.343 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: 2024-09-16T03:07:56.343 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (166 ms) 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (166 ms total) 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (48 ms) 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (23 ms) 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (72 ms total) 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: 2024-09-16T03:07:56.344 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (30 ms) 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (8 ms) 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (38 ms total) 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (62 ms) 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (3 ms) 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (65 ms total) 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-16T03:07:56.345 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-16T03:07:56.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1027 ms) 2024-09-16T03:07:56.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1027 ms total) 2024-09-16T03:07:56.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: 2024-09-16T03:07:56.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-16T03:07:56.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (114762 ms total) 2024-09-16T03:07:56.346 INFO:tasks.workunit.client.0.smithi046.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-16T03:08:06.349 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-16T03:08:06.349 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-16T03:08:06.349 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-16T03:08:06.349 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-16T03:08:06.349 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-16T03:08:06.350 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2205 ms) 2024-09-16T03:08:06.350 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-16T03:08:06.350 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (16634 ms) 2024-09-16T03:08:06.350 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-16T03:08:06.350 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5952 ms) 2024-09-16T03:08:06.351 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-16T03:08:06.351 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3018 ms) 2024-09-16T03:08:06.351 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-16T03:08:06.351 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3012 ms) 2024-09-16T03:08:06.351 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-16T03:08:06.351 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3016 ms) 2024-09-16T03:08:06.352 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-16T03:08:06.352 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2960 ms) 2024-09-16T03:08:06.352 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-16T03:08:06.352 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3017 ms) 2024-09-16T03:08:06.352 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-16T03:08:06.352 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2999 ms) 2024-09-16T03:08:06.353 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-16T03:08:06.353 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2999 ms) 2024-09-16T03:08:06.353 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-16T03:08:06.353 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2999 ms) 2024-09-16T03:08:06.353 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-16T03:08:06.353 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2994 ms) 2024-09-16T03:08:06.354 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-16T03:08:06.354 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3033 ms) 2024-09-16T03:08:06.354 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-16T03:08:06.354 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (16791 ms) 2024-09-16T03:08:06.354 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-16T03:08:06.354 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (17146 ms) 2024-09-16T03:08:06.354 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-16T03:08:06.354 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3022 ms) 2024-09-16T03:08:06.355 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-16T03:08:06.355 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3040 ms) 2024-09-16T03:08:06.355 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-16T03:08:06.355 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2987 ms) 2024-09-16T03:08:06.355 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-16T03:08:06.355 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3017 ms) 2024-09-16T03:08:06.355 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T03:08:06.356 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3025 ms) 2024-09-16T03:08:06.356 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-16T03:08:06.356 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2971 ms) 2024-09-16T03:08:06.356 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-16T03:08:06.356 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3003 ms) 2024-09-16T03:08:06.356 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-16T03:08:06.357 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2997 ms) 2024-09-16T03:08:06.357 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-16T03:08:06.357 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3010 ms) 2024-09-16T03:08:06.357 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-16T03:08:06.357 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3003 ms) 2024-09-16T03:08:06.358 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-16T03:08:06.358 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3013 ms) 2024-09-16T03:08:06.358 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-16T03:09:00.815 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T03:09:00.815 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:09:00.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:09:00.820 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:09:00.820 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:09:00.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:09:00.911 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:00.906+0000 7f2609745e00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 113 -> 0 (sync_force? mon rebuild?) 2024-09-16T03:09:00.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient: get_monmap_and_config 2024-09-16T03:09:00.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient: build_initial_monmap 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient: monmap: 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:00.942835+0000 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:00.942835+0000 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:00.944 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:00.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 1 Processor -- start 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 1 -- start start 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient: init 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient(hunting): picked mon.noname-b con 0x7f46100a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient(hunting): picked mon.noname-a con 0x7f46100a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient(hunting): picked mon.noname-c con 0x7f4610073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461009d9e0 con 0x7f46100a5810 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461009dc20 con 0x7f4610073d40 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4610073360 con 0x7f46100a4d20 2024-09-16T03:09:00.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.938+0000 7f461769b640 10 monclient(hunting): authenticate will time out at 1231.591933s 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4615437640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f46100a4d20 0x7f46100a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55042/0 (socket says 172.21.15.45:55042) 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4615437640 1 -- 172.21.15.45:0/424863721 learned_addr learned my addr 172.21.15.45:0/424863721 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1648520234 0 0) 0x7f461009dc20 con 0x7f4610073d40 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient(hunting): _negotiate 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient(hunting): my global_id is 22618 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45fc003530 con 0x7f4610073d40 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4010181257 0 0) 0x7f4610073360 con 0x7f46100a4d20 2024-09-16T03:09:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient(hunting): _negotiate 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient(hunting): my global_id is 28829 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f461009dc20 con 0x7f46100a4d20 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3275537278 0 0) 0x7f461009dc20 con 0x7f46100a4d20 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 >> v1:172.21.15.46:6789/0 conn(0x7f4610073d40 legacy=0x7f4610140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 >> v1:172.21.15.45:6789/0 conn(0x7f46100a5810 legacy=0x7f461013cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: _finish_hunting 0 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 monclient: found mon.noname-b 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461009cf30 con 0x7f46100a4d20 2024-09-16T03:09:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: _finish_auth 0 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: _check_auth_tickets 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f460c001680 con 0x7f46100a4d20 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: dump: 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:00.952 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 5 monclient: authenticate success, global_id 28829 2024-09-16T03:09:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f460c001820 con 0x7f46100a4d20 2024-09-16T03:09:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 1 -- 172.21.15.45:0/424863721 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f460c001cb0 con 0x7f46100a4d20 2024-09-16T03:09:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f4607fff640 10 monclient: dump: 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 4 set_mon_vals no callback set 2024-09-16T03:09:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 10 monclient: shutdown 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 -- 172.21.15.45:0/424863721 >> v1:172.21.15.45:6790/0 conn(0x7f46100a4d20 legacy=0x7f46100a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 -- 172.21.15.45:0/424863721 shutdown_connections 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 -- 172.21.15.45:0/424863721 >> 172.21.15.45:0/424863721 conn(0x7f46100aa720 msgr2=0x7f46100aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 -- 172.21.15.45:0/424863721 shutdown_connections 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 -- 172.21.15.45:0/424863721 wait complete. 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 10 monclient: build_initial_monmap 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 10 monclient: monmap: 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:00.955 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 librados: starting msgr at 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 librados: starting objecter 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 Processor -- start 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 -- start start 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 librados: setting wanted keys 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 1 librados: calling monclient init 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.942+0000 7f461769b640 10 monclient: init 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient(hunting): picked mon.noname-b con 0x7f4610073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient(hunting): picked mon.noname-a con 0x7f46100a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient(hunting): picked mon.noname-c con 0x7f46100a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4610140070 con 0x7f46100a4d20 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461013d440 con 0x7f4610073d40 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4610140350 con 0x7f46100a5810 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient(hunting): authenticate will time out at 1231.596170s 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4614c36640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f46100a4d20 0x7f461013cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35352/0 (socket says 172.21.15.45:35352) 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4614c36640 1 -- 172.21.15.45:0/2684329737 learned_addr learned my addr 172.21.15.45:0/2684329737 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 client.?.objecter ms_handle_connect 0x7f46100a4d20 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 client.?.objecter ms_handle_connect 0x7f46100a5810 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2096577041 0 0) 0x7f4610140350 con 0x7f46100a5810 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): my global_id is 28832 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45f4004ec0 con 0x7f46100a5810 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 client.?.objecter ms_handle_connect 0x7f4610073d40 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3850901359 0 0) 0x7f461013d440 con 0x7f4610073d40 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): my global_id is 22621 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4610140350 con 0x7f4610073d40 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1889319892 0 0) 0x7f45f4004ec0 con 0x7f46100a5810 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f461013d440 con 0x7f46100a5810 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f46080015e0 con 0x7f46100a5810 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): dump: 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2915315453 0 0) 0x7f461013d440 con 0x7f46100a5810 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 >> v1:172.21.15.46:6789/0 conn(0x7f4610073d40 legacy=0x7f461013c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 >> v1:172.21.15.45:6789/0 conn(0x7f46100a4d20 legacy=0x7f461013cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 monclient: found mon.c 2024-09-16T03:09:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46100a1510 con 0x7f46100a5810 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient: _finish_auth 0 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 5 monclient: authenticate success, global_id 28832 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient: _renew_subs 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f461013fd90 con 0x7f46100a5810 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 client.28832.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4608003980 con 0x7f46100a5810 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient: _renew_subs 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f461013ee20 con 0x7f46100a5810 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 1 librados: init done 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4608003cd0 con 0x7f46100a5810 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 monclient: dump: 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f46077fe640 4 set_mon_vals no callback set 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f461009d6d0 con 0x7f46100a5810 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f460800fd40 con 0x7f46100a5810 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(181..181 src has 1..181) ==== 5811+0+0 (unknown 3630275783 0 0) 0x7f460804a9b0 con 0x7f46100a5810 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 10 client.28832.objecter ms_dispatch 0x7f4610000ba0 osd_map(181..181 src has 1..181) 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 3 client.28832.objecter handle_osd_map got epochs [181,181] > 0 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 3 client.28832.objecter handle_osd_map decoding full epoch 181 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4605ffb640 20 client.28832.objecter dump_active .. 0 homeless 2024-09-16T03:09:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.946+0000 7f4615437640 10 monclient: get_auth_request con 0x7f45f403e250 auth_method 0 2024-09-16T03:09:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.950+0000 7f4605ffb640 10 client.28832.objecter ms_handle_connect 0x7f45f403e250 2024-09-16T03:09:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.950+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4608015a80 con 0x7f46100a5810 2024-09-16T03:09:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.950+0000 7f4605ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:00.950+0000 7f4605ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:01.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.214+0000 7f461769b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:09:01.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.214+0000 7f461769b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:01.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.214+0000 7f461769b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.214+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4610076b20 con 0x7f46100a5810 2024-09-16T03:09:01.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.214+0000 7f4605ffb640 1 -- 172.21.15.45:0/2684329737 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 641053169) 0x7f4608015380 con 0x7f46100a5810 2024-09-16T03:09:01.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.214+0000 7f4605ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:01.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.214+0000 7f4605ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:01.221 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:09:01.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 10 librados: watch_flush enter 2024-09-16T03:09:01.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 10 librados: watch_flush exit 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 20 client.28832.objecter shutdown clearing up homeless session... 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 10 client.28832.objecter successfully canceled tick 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 >> v1:172.21.15.46:6812/149360280 conn(0x7f45f403e250 legacy=0x7f45f4040710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 10 monclient: shutdown 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 >> v1:172.21.15.45:6790/0 conn(0x7f46100a5810 legacy=0x7f46101d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 shutdown_connections 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 >> 172.21.15.45:0/2684329737 conn(0x7f46100aa720 msgr2=0x7f46100a3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 shutdown_connections 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 1 -- 172.21.15.45:0/2684329737 wait complete. 2024-09-16T03:09:01.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.218+0000 7f461769b640 1 librados: shutdown 2024-09-16T03:09:01.238 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:09:01.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:09:01.238 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:09:01.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient: get_monmap_and_config 2024-09-16T03:09:01.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient: build_initial_monmap 2024-09-16T03:09:01.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient: monmap: 2024-09-16T03:09:01.362 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:01.361278+0000 2024-09-16T03:09:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:01.361278+0000 2024-09-16T03:09:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:01.363 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:01.363 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 1 Processor -- start 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 1 -- start start 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient: init 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient(hunting): picked mon.noname-c con 0x7f481c135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient(hunting): picked mon.noname-a con 0x7f481c09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient(hunting): picked mon.noname-b con 0x7f481c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481c09f7e0 con 0x7f481c09fc40 2024-09-16T03:09:01.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481c09b160 con 0x7f481c135950 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481c09b440 con 0x7f481c13c010 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient(hunting): authenticate will time out at 1232.010892s 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481ad9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f481c13c010 0x7f481c13e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55052/0 (socket says 172.21.15.45:55052) 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481ad9d640 1 -- 172.21.15.45:0/256399659 learned_addr learned my addr 172.21.15.45:0/256399659 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104027274 0 0) 0x7f481c09b440 con 0x7f481c13c010 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient(hunting): my global_id is 28835 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4808003530 con 0x7f481c13c010 2024-09-16T03:09:01.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3870586649 0 0) 0x7f481c09f7e0 con 0x7f481c09fc40 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient(hunting): my global_id is 104097 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f481c09b440 con 0x7f481c09fc40 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3543306044 0 0) 0x7f481c09b440 con 0x7f481c09fc40 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 >> v1:172.21.15.45:6790/0 conn(0x7f481c13c010 legacy=0x7f481c13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 >> v1:172.21.15.46:6789/0 conn(0x7f481c135950 legacy=0x7f481c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: _finish_hunting 0 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 monclient: found mon.noname-a 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f481c136440 con 0x7f481c09fc40 2024-09-16T03:09:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: _finish_auth 0 2024-09-16T03:09:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: _check_auth_tickets 2024-09-16T03:09:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f480c001a00 con 0x7f481c09fc40 2024-09-16T03:09:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: dump: 2024-09-16T03:09:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.376 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 5 monclient: authenticate success, global_id 104097 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f480c001c20 con 0x7f481c09fc40 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:01.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 1 -- 172.21.15.45:0/256399659 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f480c0020b0 con 0x7f481c09fc40 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f48209a7640 4 set_mon_vals no callback set 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.358+0000 7f481959a640 10 monclient: dump: 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.378 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.379 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.379 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: shutdown 2024-09-16T03:09:01.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:01.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- 172.21.15.45:0/256399659 >> v1:172.21.15.45:6789/0 conn(0x7f481c09fc40 legacy=0x7f481c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- 172.21.15.45:0/256399659 shutdown_connections 2024-09-16T03:09:01.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- 172.21.15.45:0/256399659 >> 172.21.15.45:0/256399659 conn(0x7f481c0a2bd0 msgr2=0x7f481c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:01.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- 172.21.15.45:0/256399659 shutdown_connections 2024-09-16T03:09:01.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- 172.21.15.45:0/256399659 wait complete. 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: build_initial_monmap 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: monmap: 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 librados: starting msgr at 2024-09-16T03:09:01.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 librados: starting objecter 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 Processor -- start 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- start start 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 librados: setting wanted keys 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 librados: calling monclient init 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: init 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient(hunting): picked mon.noname-c con 0x7f481c09fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient(hunting): picked mon.noname-b con 0x7f481c135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient(hunting): picked mon.noname-a con 0x7f481c13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481c09e9a0 con 0x7f481c13c010 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481c09ec80 con 0x7f481c135950 2024-09-16T03:09:01.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481c09d760 con 0x7f481c09fc40 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient(hunting): authenticate will time out at 1232.014270s 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f481a59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f481c09fc40 0x7f481c072990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55064/0 (socket says 172.21.15.45:55064) 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f481a59c640 1 -- 172.21.15.45:0/892837204 learned_addr learned my addr 172.21.15.45:0/892837204 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 client.?.objecter ms_handle_connect 0x7f481c13c010 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 client.?.objecter ms_handle_connect 0x7f481c09fc40 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2282102158 0 0) 0x7f481c09d760 con 0x7f481c09fc40 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): my global_id is 28838 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47fc004ec0 con 0x7f481c09fc40 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 client.?.objecter ms_handle_connect 0x7f481c135950 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3232659041 0 0) 0x7f481c09e9a0 con 0x7f481c13c010 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): my global_id is 104100 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f481c09d760 con 0x7f481c13c010 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 747455475 0 0) 0x7f47fc004ec0 con 0x7f481c09fc40 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f481c09e9a0 con 0x7f481c09fc40 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4804001a40 con 0x7f481c09fc40 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:01.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): dump: 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:01.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3567755871 0 0) 0x7f481c09ec80 con 0x7f481c135950 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): my global_id is 22627 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47fc004ec0 con 0x7f481c135950 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 332275201 0 0) 0x7f481c09d760 con 0x7f481c13c010 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f481c09ec80 con 0x7f481c13c010 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4814001140 con 0x7f481c13c010 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient(hunting): dump: 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.385 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 276077565 0 0) 0x7f481c09e9a0 con 0x7f481c09fc40 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.46:6789/0 conn(0x7f481c135950 legacy=0x7f481c0730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f481c13c010 legacy=0x7f481c1d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 monclient: found mon.c 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f481c09e290 con 0x7f481c09fc40 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient: _finish_auth 0 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 5 monclient: authenticate success, global_id 28838 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: _renew_subs 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f481c0737b0 con 0x7f481c09fc40 2024-09-16T03:09:01.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 client.28838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: _renew_subs 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f481c1d62c0 con 0x7f481c09fc40 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f48209a7640 1 librados: init done 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4804001cc0 con 0x7f481c09fc40 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4804003a70 con 0x7f481c09fc40 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4818d99640 4 set_mon_vals no callback set 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.362+0000 7f4812ffd640 10 monclient: dump: 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.388 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f48209a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:01.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48000065d0 con 0x7f4800002870 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 client.28838.objecter ms_handle_connect 0x7f4800002870 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 client.28838.objecter resend_mon_ops 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4804003c10 con 0x7f481c09fc40 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(182..182 src has 1..182) ==== 5522+0+0 (unknown 2666811546 0 0) 0x7f480404b860 con 0x7f481c09fc40 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 client.28838.objecter ms_dispatch 0x7f481c000ba0 osd_map(182..182 src has 1..182) 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 3 client.28838.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 3 client.28838.objecter handle_osd_map decoding full epoch 182 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 20 client.28838.objecter dump_active .. 0 homeless 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902204592 0 0) 0x7f48000065d0 con 0x7f4800002870 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 monclient(hunting): my global_id is 104103 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f481c09e9a0 con 0x7f4800002870 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f481ad9d640 10 monclient: get_auth_request con 0x7f47fc03e390 auth_method 0 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3640502726 0 0) 0x7f481c09e9a0 con 0x7f4800002870 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48000065d0 con 0x7f4800002870 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f480c001a80 con 0x7f4800002870 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 monclient: dump: 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.391 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 697022117 0 0) 0x7f48000065d0 con 0x7f4800002870 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f480000a170 con 0x7f4800002870 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 client.28838.objecter ms_handle_connect 0x7f47fc03e390 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f480c001d80 con 0x7f4800002870 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f4800002870 legacy=0x7f4800004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.366+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f4800002870 legacy=0x7f4800004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:01.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.370+0000 7f48209a7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:09:01.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.370+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4800009e80 con 0x7f4800009140 2024-09-16T03:09:01.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.370+0000 7f4812ffd640 10 client.28838.objecter ms_handle_connect 0x7f4800009140 2024-09-16T03:09:01.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.370+0000 7f4812ffd640 10 client.28838.objecter resend_mon_ops 2024-09-16T03:09:01.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2979137214 0 0) 0x7f4800009e80 con 0x7f4800009140 2024-09-16T03:09:01.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient(hunting): my global_id is 104106 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48000065d0 con 0x7f4800009140 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2846427304 0 0) 0x7f48000065d0 con 0x7f4800009140 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4800009e80 con 0x7f4800009140 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f480c003b20 con 0x7f4800009140 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient: dump: 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.396 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2148883062 0 0) 0x7f4800009e80 con 0x7f4800009140 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f480000c420 con 0x7f4800009140 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4819d9b640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f4800009140 legacy=0x7f48000095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4819d9b640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f4800009140 legacy=0x7f48000095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:09:01.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4819d9b640 1 --1- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f4800009140 0x7f48000095d0 :-1 s=OPENED pgs=50 cs=1 l=1).handle_message read tag failed 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4819d9b640 1 --1- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f4800009140 0x7f48000095d0 :-1 s=OPENED pgs=50 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f4800009140 legacy=0x7f48000095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f4800009140 legacy=0x7f48000095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4800009e80 con 0x7f47fc049340 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 client.28838.objecter ms_handle_connect 0x7f47fc049340 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 client.28838.objecter resend_mon_ops 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2088940315 0 0) 0x7f4800009e80 con 0x7f47fc049340 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient(hunting): my global_id is 104109 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48000065d0 con 0x7f47fc049340 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 814250205 0 0) 0x7f48000065d0 con 0x7f47fc049340 2024-09-16T03:09:01.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4800009e80 con 0x7f47fc049340 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f480c0050f0 con 0x7f47fc049340 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient: dump: 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.399 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2075513350 0 0) 0x7f4800009e80 con 0x7f47fc049340 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f480c001d80 con 0x7f47fc049340 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3967 (unknown 0 0 1134820998) 0x7f480c0052b0 con 0x7f47fc049340 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f47fc049340 legacy=0x7f47fc047d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f4812ffd640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6789/0 conn(0x7f47fc049340 legacy=0x7f47fc047d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f481ad9d640 1 -- 172.21.15.45:0/892837204 reap_dead start 2024-09-16T03:09:01.400 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 20, 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 475, 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:09:01.401 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:01.402 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:01.403 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:09:01.404 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:09:01.405 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:09:01.405 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:09:01.405 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:09:01.405 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:09:01.405 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:01.405 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:09:01.405 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:01.405 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:09:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:09:01.412 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:09:01.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 10 librados: watch_flush enter 2024-09-16T03:09:01.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 10 librados: watch_flush exit 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 20 client.28838.objecter shutdown clearing up homeless session... 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 10 client.28838.objecter successfully canceled tick 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.46:6812/149360280 conn(0x7f47fc03e390 legacy=0x7f47fc040850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 10 monclient: shutdown 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 >> v1:172.21.15.45:6790/0 conn(0x7f481c09fc40 legacy=0x7f481c072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 shutdown_connections 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 >> 172.21.15.45:0/892837204 conn(0x7f481c0a2bd0 msgr2=0x7f481c13ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 shutdown_connections 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 1 -- 172.21.15.45:0/892837204 wait complete. 2024-09-16T03:09:01.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.374+0000 7f48209a7640 1 librados: shutdown 2024-09-16T03:09:01.415 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:09:01.415 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:09:01.415 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:09:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient: get_monmap_and_config 2024-09-16T03:09:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient: build_initial_monmap 2024-09-16T03:09:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient: monmap: 2024-09-16T03:09:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:01.530 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.530 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:01.528639+0000 2024-09-16T03:09:01.530 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:01.528639+0000 2024-09-16T03:09:01.530 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:01.530 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.530 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:01.530 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:01.530 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:01.530 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 Processor -- start 2024-09-16T03:09:01.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 -- start start 2024-09-16T03:09:01.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient: init 2024-09-16T03:09:01.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:01.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient(hunting): picked mon.noname-b con 0x7fc0040a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient(hunting): picked mon.noname-a con 0x7fc0040a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient(hunting): picked mon.noname-c con 0x7fc004073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00409d9e0 con 0x7fc0040a5810 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00409dc20 con 0x7fc004073d40 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc004073360 con 0x7fc0040a4d20 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient(hunting): authenticate will time out at 1232.177703s 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc009856640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc0040a4d20 0x7fc0040a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55074/0 (socket says 172.21.15.45:55074) 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc009856640 1 -- 172.21.15.45:0/1084785229 learned_addr learned my addr 172.21.15.45:0/1084785229 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1601379877 0 0) 0x7fc004073360 con 0x7fc0040a4d20 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): my global_id is 28841 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff0003530 con 0x7fc0040a4d20 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2776102508 0 0) 0x7fc00409d9e0 con 0x7fc0040a5810 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): my global_id is 104112 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc004073360 con 0x7fc0040a5810 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4019203460 0 0) 0x7fc00409dc20 con 0x7fc004073d40 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient(hunting): my global_id is 22630 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc00409d9e0 con 0x7fc004073d40 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3846650430 0 0) 0x7fbff0003530 con 0x7fc0040a4d20 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 >> v1:172.21.15.46:6789/0 conn(0x7fc004073d40 legacy=0x7fc004140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 >> v1:172.21.15.45:6789/0 conn(0x7fc0040a5810 legacy=0x7fc00413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: _finish_hunting 0 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 monclient: found mon.noname-b 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00409cf30 con 0x7fc0040a4d20 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: _finish_auth 0 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: _check_auth_tickets 2024-09-16T03:09:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbff40016a0 con 0x7fc0040a4d20 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 5 monclient: authenticate success, global_id 28841 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: dump: 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbff40018c0 con 0x7fc0040a4d20 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 1 -- 172.21.15.45:0/1084785229 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbff4001d10 con 0x7fc0040a4d20 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc008854640 10 monclient: dump: 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.538 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 4 set_mon_vals no callback set 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 10 monclient: shutdown 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 -- 172.21.15.45:0/1084785229 >> v1:172.21.15.45:6790/0 conn(0x7fc0040a4d20 legacy=0x7fc0040a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 -- 172.21.15.45:0/1084785229 shutdown_connections 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 -- 172.21.15.45:0/1084785229 >> 172.21.15.45:0/1084785229 conn(0x7fc0040aa720 msgr2=0x7fc0040aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 -- 172.21.15.45:0/1084785229 shutdown_connections 2024-09-16T03:09:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.526+0000 7fc00baba640 1 -- 172.21.15.45:0/1084785229 wait complete. 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient: build_initial_monmap 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient: monmap: 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 librados: starting msgr at 2024-09-16T03:09:01.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 librados: starting objecter 2024-09-16T03:09:01.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 Processor -- start 2024-09-16T03:09:01.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 -- start start 2024-09-16T03:09:01.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 librados: setting wanted keys 2024-09-16T03:09:01.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 librados: calling monclient init 2024-09-16T03:09:01.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient: init 2024-09-16T03:09:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient(hunting): picked mon.noname-b con 0x7fc004073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient(hunting): picked mon.noname-a con 0x7fc0040a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient(hunting): picked mon.noname-c con 0x7fc0040a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0041406f0 con 0x7fc0040a4d20 2024-09-16T03:09:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00413d6f0 con 0x7fc004073d40 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0040a1f60 con 0x7fc0040a5810 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient(hunting): authenticate will time out at 1232.181151s 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc009055640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc0040a4d20 0x7fc00413cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35410/0 (socket says 172.21.15.45:35410) 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc009055640 1 -- 172.21.15.45:0/1488058623 learned_addr learned my addr 172.21.15.45:0/1488058623 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00a057640 1 --1- 172.21.15.45:0/1488058623 >> v1:172.21.15.45:6790/0 conn(0x7fc0040a5810 0x7fc0041d4fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55088/0 (socket says 172.21.15.45:55088) 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 client.?.objecter ms_handle_connect 0x7fc0040a5810 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 client.?.objecter ms_handle_connect 0x7fc0040a4d20 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1387612903 0 0) 0x7fc0041406f0 con 0x7fc0040a4d20 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): my global_id is 104115 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe8004ec0 con 0x7fc0040a4d20 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 649588207 0 0) 0x7fc0040a1f60 con 0x7fc0040a5810 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): my global_id is 28844 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0041406f0 con 0x7fc0040a5810 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 client.?.objecter ms_handle_connect 0x7fc004073d40 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 348760891 0 0) 0x7fbfe8004ec0 con 0x7fc0040a4d20 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0040a1f60 con 0x7fc0040a4d20 2024-09-16T03:09:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc000001940 con 0x7fc0040a4d20 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): dump: 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1726743852 0 0) 0x7fc0041406f0 con 0x7fc0040a5810 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfe8004ec0 con 0x7fc0040a5810 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2287059194 0 0) 0x7fc00413d6f0 con 0x7fc004073d40 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): my global_id is 22633 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0041406f0 con 0x7fc004073d40 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbffc001c80 con 0x7fc0040a5810 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): dump: 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.550 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 819227315 0 0) 0x7fc0040a1f60 con 0x7fc0040a4d20 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 >> v1:172.21.15.45:6790/0 conn(0x7fc0040a5810 legacy=0x7fc0041d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 >> v1:172.21.15.46:6789/0 conn(0x7fc004073d40 legacy=0x7fc00413c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 monclient: found mon.a 2024-09-16T03:09:01.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0040a1510 con 0x7fc0040a4d20 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: _finish_auth 0 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 5 monclient: authenticate success, global_id 104115 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient: _renew_subs 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc00413e390 con 0x7fc0040a4d20 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 client.104115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient: _renew_subs 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc004138a30 con 0x7fc0040a4d20 2024-09-16T03:09:01.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 librados: init done 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc000003620 con 0x7fc0040a4d20 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc0000039f0 con 0x7fc0040a4d20 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: dump: 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffbfff640 4 set_mon_vals no callback set 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbfec0065d0 con 0x7fbfec002870 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 client.104115.objecter ms_handle_connect 0x7fbfec002870 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 client.104115.objecter resend_mon_ops 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fc000003b90 con 0x7fc0040a4d20 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(182..182 src has 1..182) ==== 5522+0+0 (unknown 2666811546 0 0) 0x7fc00004b6e0 con 0x7fc0040a4d20 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 client.104115.objecter ms_dispatch 0x7fc004000ba0 osd_map(182..182 src has 1..182) 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 3 client.104115.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 3 client.104115.objecter handle_osd_map decoding full epoch 182 2024-09-16T03:09:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 20 client.104115.objecter dump_active .. 0 homeless 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4120366436 0 0) 0x7fbfec0065d0 con 0x7fbfec002870 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient(hunting): my global_id is 28847 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0040a1f60 con 0x7fbfec002870 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fc00a057640 10 monclient: get_auth_request con 0x7fbfe803e1f0 auth_method 0 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3861825726 0 0) 0x7fc0040a1f60 con 0x7fbfec002870 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfec0065d0 con 0x7fbfec002870 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbff4001660 con 0x7fbfec002870 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.530+0000 7fbffa7fc640 10 monclient: dump: 2024-09-16T03:09:01.555 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.534+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 564621443 0 0) 0x7fbfec0065d0 con 0x7fbfec002870 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.534+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfec00a170 con 0x7fbfec002870 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.534+0000 7fbffa7fc640 10 client.104115.objecter ms_handle_connect 0x7fbfe803e1f0 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.534+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbff40049d0 con 0x7fbfec002870 2024-09-16T03:09:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.534+0000 7fbffa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.534+0000 7fbffa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.534+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 >> v1:172.21.15.45:6790/0 conn(0x7fbfec002870 legacy=0x7fbfec004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.534+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 >> v1:172.21.15.45:6790/0 conn(0x7fbfec002870 legacy=0x7fbfec004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbfec009e80 con 0x7fbfec009140 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 client.104115.objecter ms_handle_connect 0x7fbfec009140 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 client.104115.objecter resend_mon_ops 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 644638042 0 0) 0x7fbfec009e80 con 0x7fbfec009140 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 monclient(hunting): my global_id is 28850 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfec0065d0 con 0x7fbfec009140 2024-09-16T03:09:01.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3444828926 0 0) 0x7fbfec0065d0 con 0x7fbfec009140 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfec009e80 con 0x7fbfec009140 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbff4001bc0 con 0x7fbfec009140 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 monclient: dump: 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.559 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 210151361 0 0) 0x7fbfec009e80 con 0x7fbfec009140 2024-09-16T03:09:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbfec00c420 con 0x7fbfec009140 2024-09-16T03:09:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 3555994869) 0x7fbff4001b00 con 0x7fbfec009140 2024-09-16T03:09:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:09:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 >> v1:172.21.15.45:6790/0 conn(0x7fbfec009140 legacy=0x7fbfec0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fbffa7fc640 1 -- 172.21.15.45:0/1488058623 >> v1:172.21.15.45:6790/0 conn(0x7fbfec009140 legacy=0x7fbfec0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:01.559 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 20, 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 263391, 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:09:01.560 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.561 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:09:01.561 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:01.561 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:09:01.561 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:09:01.561 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:01.561 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:01.561 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:01.561 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:01.561 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:01.562 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:09:01.563 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:01.564 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.565 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.566 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:01.567 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.568 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:01.569 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:09:01.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.572 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:09:01.572 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:09:01.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 10 librados: watch_flush enter 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 10 librados: watch_flush exit 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 20 client.104115.objecter shutdown clearing up homeless session... 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 10 client.104115.objecter successfully canceled tick 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 >> v1:172.21.15.46:6812/149360280 conn(0x7fbfe803e1f0 legacy=0x7fbfe80406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 10 monclient: shutdown 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 >> v1:172.21.15.45:6789/0 conn(0x7fc0040a4d20 legacy=0x7fc00413cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00a057640 1 -- 172.21.15.45:0/1488058623 reap_dead start 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00a057640 1 -- 172.21.15.45:0/1488058623 reap_dead start 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 shutdown_connections 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.538+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 >> 172.21.15.45:0/1488058623 conn(0x7fc0040aa720 msgr2=0x7fc0040a3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.542+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 shutdown_connections 2024-09-16T03:09:01.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.542+0000 7fc00baba640 1 -- 172.21.15.45:0/1488058623 wait complete. 2024-09-16T03:09:01.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.542+0000 7fc00baba640 1 librados: shutdown 2024-09-16T03:09:01.575 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:09:01.575 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:09:01.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:09:01.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient: get_monmap_and_config 2024-09-16T03:09:01.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient: build_initial_monmap 2024-09-16T03:09:01.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient: monmap: 2024-09-16T03:09:01.693 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:01.693 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.694 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:01.692544+0000 2024-09-16T03:09:01.694 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:01.692544+0000 2024-09-16T03:09:01.694 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:01.694 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.694 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:01.694 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:01.694 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:01.694 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 Processor -- start 2024-09-16T03:09:01.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 -- start start 2024-09-16T03:09:01.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient: init 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient(hunting): picked mon.noname-c con 0x7fe108135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient(hunting): picked mon.noname-b con 0x7fe10809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient(hunting): picked mon.noname-a con 0x7fe10813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe10809f7e0 con 0x7fe10813c010 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe10809b160 con 0x7fe108135950 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe10809b440 con 0x7fe10809fc40 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient(hunting): authenticate will time out at 1232.341905s 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe107fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe10809fc40 0x7fe10813a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55110/0 (socket says 172.21.15.45:55110) 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10d3c1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe10813c010 0x7fe10813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35412/0 (socket says 172.21.15.45:35412) 2024-09-16T03:09:01.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe107fff640 1 -- 172.21.15.45:0/2529837203 learned_addr learned my addr 172.21.15.45:0/2529837203 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4240758570 0 0) 0x7fe10809b440 con 0x7fe10809fc40 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient(hunting): my global_id is 28853 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0f4003530 con 0x7fe10809fc40 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1477790172 0 0) 0x7fe10809f7e0 con 0x7fe10813c010 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient(hunting): my global_id is 104118 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe10809b440 con 0x7fe10813c010 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10d3c1640 1 -- 172.21.15.45:0/2529837203 >> v1:172.21.15.45:6789/0 conn(0x7fe10813c010 legacy=0x7fe10813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T03:09:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10d3c1640 1 -- 172.21.15.45:0/2529837203 >> v1:172.21.15.45:6789/0 conn(0x7fe10813c010 legacy=0x7fe10813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10d3c1640 1 --1- 172.21.15.45:0/2529837203 >> v1:172.21.15.45:6789/0 conn(0x7fe10813c010 0x7fe10813e400 :-1 s=OPENED pgs=54 cs=1 l=1).handle_message read tag failed 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10d3c1640 1 --1- 172.21.15.45:0/2529837203 >> v1:172.21.15.45:6789/0 conn(0x7fe10813c010 0x7fe10813e400 :-1 s=OPENED pgs=54 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.45:6789/0 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3688381968 0 0) 0x7fe0f4003530 con 0x7fe10809fc40 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 >> v1:172.21.15.46:6789/0 conn(0x7fe108135950 legacy=0x7fe108135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 >> v1:172.21.15.45:6789/0 conn(0x7fe10813c010 legacy=0x7fe10813e400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: _finish_hunting 0 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 monclient: found mon.noname-b 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe108136440 con 0x7fe10809fc40 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: _finish_auth 0 2024-09-16T03:09:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: _check_auth_tickets 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe0f00016a0 con 0x7fe10809fc40 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 5 monclient: authenticate success, global_id 28853 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: dump: 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe0f0001840 con 0x7fe10809fc40 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 1 -- 172.21.15.45:0/2529837203 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe0f0001cd0 con 0x7fe10809fc40 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 4 set_mon_vals no callback set 2024-09-16T03:09:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe1077fe640 10 monclient: dump: 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 10 monclient: shutdown 2024-09-16T03:09:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 -- 172.21.15.45:0/2529837203 >> v1:172.21.15.45:6790/0 conn(0x7fe10809fc40 legacy=0x7fe10813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 -- 172.21.15.45:0/2529837203 shutdown_connections 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 -- 172.21.15.45:0/2529837203 >> 172.21.15.45:0/2529837203 conn(0x7fe1080a2bd0 msgr2=0x7fe1080a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 -- 172.21.15.45:0/2529837203 shutdown_connections 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.690+0000 7fe10ee24640 1 -- 172.21.15.45:0/2529837203 wait complete. 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient: build_initial_monmap 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient: monmap: 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.708 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 librados: starting msgr at 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 librados: starting objecter 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 Processor -- start 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 -- start start 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 librados: setting wanted keys 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 librados: calling monclient init 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient: init 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient(hunting): picked mon.noname-b con 0x7fe10809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient(hunting): picked mon.noname-a con 0x7fe108135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient(hunting): picked mon.noname-c con 0x7fe10813c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1080718d0 con 0x7fe108135950 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe108071bb0 con 0x7fe10809fc40 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe10806f1d0 con 0x7fe10813c010 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient(hunting): authenticate will time out at 1232.345129s 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe107fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe108135950 0x7fe10806eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35414/0 (socket says 172.21.15.45:35414) 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe107fff640 1 -- 172.21.15.45:0/1207182456 learned_addr learned my addr 172.21.15.45:0/1207182456 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 client.?.objecter ms_handle_connect 0x7fe108135950 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 client.?.objecter ms_handle_connect 0x7fe10813c010 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1407102556 0 0) 0x7fe1080718d0 con 0x7fe108135950 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): my global_id is 104121 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0e8004ec0 con 0x7fe108135950 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3390453676 0 0) 0x7fe10806f1d0 con 0x7fe10813c010 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): my global_id is 28856 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1080718d0 con 0x7fe10813c010 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 client.?.objecter ms_handle_connect 0x7fe10809fc40 2024-09-16T03:09:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1555817369 0 0) 0x7fe1080718d0 con 0x7fe10813c010 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe10806f1d0 con 0x7fe10813c010 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe1000019f0 con 0x7fe10813c010 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): dump: 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.712 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 79714360 0 0) 0x7fe108071bb0 con 0x7fe10809fc40 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): my global_id is 22639 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1080718d0 con 0x7fe10809fc40 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4215695327 0 0) 0x7fe0e8004ec0 con 0x7fe108135950 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe108071bb0 con 0x7fe108135950 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe0f0001b90 con 0x7fe108135950 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:01.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:01.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): dump: 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1144385346 0 0) 0x7fe10806f1d0 con 0x7fe10813c010 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 >> v1:172.21.15.46:6789/0 conn(0x7fe10809fc40 legacy=0x7fe10806e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 >> v1:172.21.15.45:6789/0 conn(0x7fe108135950 legacy=0x7fe10806eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient: _finish_hunting 0 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 monclient: found mon.c 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1080742d0 con 0x7fe10813c010 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient: _finish_auth 0 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient: _check_auth_tickets 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 5 monclient: authenticate success, global_id 28856 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient: _renew_subs 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe108071310 con 0x7fe10813c010 2024-09-16T03:09:01.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 client.28856.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient: _renew_subs 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe10806f8d0 con 0x7fe10813c010 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 librados: init done 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1000047f0 con 0x7fe10813c010 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe100004bc0 con 0x7fe10813c010 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient: dump: 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe106ffd640 4 set_mon_vals no callback set 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe0ec0065d0 con 0x7fe0ec002870 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fe100004d60 con 0x7fe10813c010 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(182..182 src has 1..182) ==== 5522+0+0 (unknown 2666811546 0 0) 0x7fe10004b690 con 0x7fe10813c010 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 client.28856.objecter ms_dispatch 0x7fe108000ba0 osd_map(182..182 src has 1..182) 2024-09-16T03:09:01.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 3 client.28856.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 3 client.28856.objecter handle_osd_map decoding full epoch 182 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 20 client.28856.objecter dump_active .. 0 homeless 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 client.28856.objecter ms_handle_connect 0x7fe0ec002870 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 client.28856.objecter resend_mon_ops 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe107fff640 10 monclient: get_auth_request con 0x7fe0e803e300 auth_method 0 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352730022 0 0) 0x7fe0ec0065d0 con 0x7fe0ec002870 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 10 monclient(hunting): my global_id is 22642 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.694+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe10806f1d0 con 0x7fe0ec002870 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 10 client.28856.objecter ms_handle_connect 0x7fe0e803e300 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 527901627 0 0) 0x7fe10806f1d0 con 0x7fe0ec002870 2024-09-16T03:09:01.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0ec0065d0 con 0x7fe0ec002870 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe0f8001b10 con 0x7fe0ec002870 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 10 monclient: dump: 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.722 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3041953067 0 0) 0x7fe0ec0065d0 con 0x7fe0ec002870 2024-09-16T03:09:01.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0ec00a170 con 0x7fe0ec002870 2024-09-16T03:09:01.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe0f8001d10 con 0x7fe0ec002870 2024-09-16T03:09:01.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:01.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:01.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 >> v1:172.21.15.46:6789/0 conn(0x7fe0ec002870 legacy=0x7fe0ec004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.698+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 >> v1:172.21.15.46:6789/0 conn(0x7fe0ec002870 legacy=0x7fe0ec004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.702+0000 7fe10ee24640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.702+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe0ec009e80 con 0x7fe0ec009140 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 client.28856.objecter ms_handle_connect 0x7fe0ec009140 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 client.28856.objecter resend_mon_ops 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028040550 0 0) 0x7fe0ec009e80 con 0x7fe0ec009140 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 monclient(hunting): my global_id is 22645 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0ec0065d0 con 0x7fe0ec009140 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2126589191 0 0) 0x7fe0ec0065d0 con 0x7fe0ec009140 2024-09-16T03:09:01.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0ec009e80 con 0x7fe0ec009140 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe0f8003b20 con 0x7fe0ec009140 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 monclient: dump: 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.726 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 576689647 0 0) 0x7fe0ec009e80 con 0x7fe0ec009140 2024-09-16T03:09:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe0ec00c420 con 0x7fe0ec009140 2024-09-16T03:09:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 1281029508) 0x7fe0f8003ab0 con 0x7fe0ec009140 2024-09-16T03:09:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:09:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 >> v1:172.21.15.46:6789/0 conn(0x7fe0ec009140 legacy=0x7fe0ec0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe1057fa640 1 -- 172.21.15.45:0/1207182456 >> v1:172.21.15.46:6789/0 conn(0x7fe0ec009140 legacy=0x7fe0ec0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:01.727 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:09:01.727 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 20, 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 263512, 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:09:01.728 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.729 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:09:01.731 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:01.732 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.733 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:01.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.735 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:01.736 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.737 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:01.739 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:01.739 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:01.739 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:01.739 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:01.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:01.739 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:09:01.739 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 10 librados: watch_flush enter 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 10 librados: watch_flush exit 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 20 client.28856.objecter shutdown clearing up homeless session... 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 10 client.28856.objecter successfully canceled tick 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 >> v1:172.21.15.46:6812/149360280 conn(0x7fe0e803e300 legacy=0x7fe0e80407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 10 monclient: shutdown 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 >> v1:172.21.15.45:6790/0 conn(0x7fe10813c010 legacy=0x7fe1081d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10d3c1640 1 -- 172.21.15.45:0/1207182456 reap_dead start 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10d3c1640 1 -- 172.21.15.45:0/1207182456 reap_dead start 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 shutdown_connections 2024-09-16T03:09:01.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 >> 172.21.15.45:0/1207182456 conn(0x7fe1080a2bd0 msgr2=0x7fe10813ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 shutdown_connections 2024-09-16T03:09:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 1 -- 172.21.15.45:0/1207182456 wait complete. 2024-09-16T03:09:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.706+0000 7fe10ee24640 1 librados: shutdown 2024-09-16T03:09:01.742 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:09:01.742 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:09:01.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.862+0000 7fcd697a3640 10 monclient: get_monmap_and_config 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.862+0000 7fcd697a3640 10 monclient: build_initial_monmap 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.862+0000 7fcd697a3640 10 monclient: monmap: 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:01.867091+0000 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:01.867091+0000 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:01.869 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:01.869 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.862+0000 7fcd697a3640 1 Processor -- start 2024-09-16T03:09:01.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.862+0000 7fcd697a3640 1 -- start start 2024-09-16T03:09:01.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.862+0000 7fcd697a3640 10 monclient: init 2024-09-16T03:09:01.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:01.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:09:01.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): picked mon.noname-b con 0x7fcd64137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:01.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): picked mon.noname-a con 0x7fcd64138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): picked mon.noname-c con 0x7fcd6413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd640a4f90 con 0x7fcd64138100 2024-09-16T03:09:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd640a4cb0 con 0x7fcd6413e2a0 2024-09-16T03:09:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd6409dc70 con 0x7fcd64137d20 2024-09-16T03:09:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): authenticate will time out at 1232.516574s 2024-09-16T03:09:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd627fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcd64138100 0x7fcd6413cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35418/0 (socket says 172.21.15.45:35418) 2024-09-16T03:09:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd627fc640 1 -- 172.21.15.45:0/650341332 learned_addr learned my addr 172.21.15.45:0/650341332 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd62ffd640 1 --1- 172.21.15.45:0/650341332 >> v1:172.21.15.45:6790/0 conn(0x7fcd64137d20 0x7fcd640a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55130/0 (socket says 172.21.15.45:55130) 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 104218177 0 0) 0x7fcd640a4f90 con 0x7fcd64138100 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient(hunting): my global_id is 104124 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd4c003530 con 0x7fcd64138100 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551319977 0 0) 0x7fcd6409dc70 con 0x7fcd64137d20 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient(hunting): my global_id is 28859 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd640a4f90 con 0x7fcd64137d20 2024-09-16T03:09:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1251342170 0 0) 0x7fcd4c003530 con 0x7fcd64138100 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 >> v1:172.21.15.45:6790/0 conn(0x7fcd64137d20 legacy=0x7fcd640a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 >> v1:172.21.15.46:6789/0 conn(0x7fcd6413e2a0 legacy=0x7fcd64140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 monclient: found mon.noname-a 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6409cc50 con 0x7fcd64138100 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: _finish_auth 0 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd580016a0 con 0x7fcd64138100 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 5 monclient: authenticate success, global_id 104124 2024-09-16T03:09:01.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: dump: 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd58001840 con 0x7fcd64138100 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 1 -- 172.21.15.45:0/650341332 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd58001cd0 con 0x7fcd64138100 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd61ffb640 10 monclient: dump: 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 4 set_mon_vals no callback set 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient: shutdown 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- 172.21.15.45:0/650341332 >> v1:172.21.15.45:6789/0 conn(0x7fcd64138100 legacy=0x7fcd6413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- 172.21.15.45:0/650341332 shutdown_connections 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- 172.21.15.45:0/650341332 >> 172.21.15.45:0/650341332 conn(0x7fcd640aa110 msgr2=0x7fcd640aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- 172.21.15.45:0/650341332 shutdown_connections 2024-09-16T03:09:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- 172.21.15.45:0/650341332 wait complete. 2024-09-16T03:09:01.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient: build_initial_monmap 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient: monmap: 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 librados: starting msgr at 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 librados: starting objecter 2024-09-16T03:09:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 Processor -- start 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- start start 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 librados: setting wanted keys 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 librados: calling monclient init 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient: init 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): picked mon.noname-a con 0x7fcd64137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): picked mon.noname-c con 0x7fcd64138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): picked mon.noname-b con 0x7fcd6413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd640a2960 con 0x7fcd64137d20 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd641418b0 con 0x7fcd6413e2a0 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd64138af0 con 0x7fcd64138100 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd697a3640 10 monclient(hunting): authenticate will time out at 1232.519873s 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd627fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fcd64138100 0x7fcd640a1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55138/0 (socket says 172.21.15.45:55138) 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.866+0000 7fcd627fc640 1 -- 172.21.15.45:0/2211232190 learned_addr learned my addr 172.21.15.45:0/2211232190 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 client.?.objecter ms_handle_connect 0x7fcd64138100 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 client.?.objecter ms_handle_connect 0x7fcd64137d20 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2064490949 0 0) 0x7fcd64138af0 con 0x7fcd64138100 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): my global_id is 28862 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd50004ec0 con 0x7fcd64138100 2024-09-16T03:09:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 client.?.objecter ms_handle_connect 0x7fcd6413e2a0 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3533240486 0 0) 0x7fcd640a2960 con 0x7fcd64137d20 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): my global_id is 104127 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd64138af0 con 0x7fcd64137d20 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4099302113 0 0) 0x7fcd50004ec0 con 0x7fcd64138100 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd640a2960 con 0x7fcd64138100 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd58001b90 con 0x7fcd64138100 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): dump: 2024-09-16T03:09:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728908365 0 0) 0x7fcd641418b0 con 0x7fcd6413e2a0 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): my global_id is 22651 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd50004ec0 con 0x7fcd6413e2a0 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2736567676 0 0) 0x7fcd64138af0 con 0x7fcd64137d20 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd641418b0 con 0x7fcd64137d20 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd5c001b60 con 0x7fcd64137d20 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient(hunting): dump: 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3946972172 0 0) 0x7fcd640a2960 con 0x7fcd64138100 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 >> v1:172.21.15.46:6789/0 conn(0x7fcd6413e2a0 legacy=0x7fcd641d4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 >> v1:172.21.15.45:6789/0 conn(0x7fcd64137d20 legacy=0x7fcd640a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 monclient: found mon.c 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd640a2120 con 0x7fcd64138100 2024-09-16T03:09:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 5 monclient: authenticate success, global_id 28862 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 10 monclient: _renew_subs 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcd640a3020 con 0x7fcd64138100 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd58004c00 con 0x7fcd64138100 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 10 client.28862.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fcd58004f50 con 0x7fcd64138100 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 10 monclient: _renew_subs 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcd64140d80 con 0x7fcd64138100 2024-09-16T03:09:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 1 librados: init done 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd617fa640 4 set_mon_vals no callback set 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 monclient: dump: 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:01.890 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fcd58010fc0 con 0x7fcd64138100 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd640a04d0 con 0x7fcd64138100 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(182..182 src has 1..182) ==== 5522+0+0 (unknown 2666811546 0 0) 0x7fcd5804b7e0 con 0x7fcd64138100 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 client.28862.objecter ms_dispatch 0x7fcd64000ba0 osd_map(182..182 src has 1..182) 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 3 client.28862.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 3 client.28862.objecter handle_osd_map decoding full epoch 182 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 20 client.28862.objecter dump_active .. 0 homeless 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd62ffd640 10 monclient: get_auth_request con 0x7fcd5003e300 auth_method 0 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.870+0000 7fcd4b7fe640 10 client.28862.objecter ms_handle_connect 0x7fcd5003e300 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.874+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fcd58016a90 con 0x7fcd64138100 2024-09-16T03:09:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.874+0000 7fcd4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:01.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:01.874+0000 7fcd4b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:02.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.114+0000 7fcd697a3640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:09:02.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.114+0000 7fcd697a3640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:09:02.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.114+0000 7fcd697a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:02.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.114+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fcd640767c0 con 0x7fcd64138100 2024-09-16T03:09:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.118+0000 7fcd4b7fe640 1 -- 172.21.15.45:0/2211232190 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fcd58016390 con 0x7fcd64138100 2024-09-16T03:09:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.118+0000 7fcd4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:09:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.118+0000 7fcd4b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:02.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 10 librados: watch_flush enter 2024-09-16T03:09:02.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 10 librados: watch_flush exit 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 20 client.28862.objecter shutdown clearing up homeless session... 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 10 client.28862.objecter successfully canceled tick 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 >> v1:172.21.15.46:6812/149360280 conn(0x7fcd5003e300 legacy=0x7fcd500407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 10 monclient: shutdown 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 >> v1:172.21.15.45:6790/0 conn(0x7fcd64138100 legacy=0x7fcd640a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 shutdown_connections 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 >> 172.21.15.45:0/2211232190 conn(0x7fcd640aa110 msgr2=0x7fcd64131130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 shutdown_connections 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 1 -- 172.21.15.45:0/2211232190 wait complete. 2024-09-16T03:09:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:02.122+0000 7fcd697a3640 1 librados: shutdown 2024-09-16T03:09:02.142 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.130+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.142 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.130+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=14,logm=58} crc {auth=2659646400,config=1097430493,health=2589925471,kv=3374818887,logm=3361592890}) 2024-09-16T03:09:02.142 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.130+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=1,health=12,kv=14,logm=57} crc {auth=244620561,config=1228700967,health=2589925471,kv=3374818887,logm=293516592}) 2024-09-16T03:09:02.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.130+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.130+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=14,logm=58} crc {auth=2659646400,config=1097430493,health=2589925471,kv=3374818887,logm=3361592890}) 2024-09-16T03:09:02.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.130+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=11,kv=14,logm=56} crc {auth=3517747360,config=1097430493,health=3864720681,kv=2462494327,logm=2024787909}) 2024-09-16T03:09:02.150 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:09:02.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.138+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.138+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059948637}) 2024-09-16T03:09:02.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.138+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=659477170}) 2024-09-16T03:09:02.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.138+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.138+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059948637}) 2024-09-16T03:09:02.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.138+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1545933850}) 2024-09-16T03:09:02.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.146+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.146+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956706933}) 2024-09-16T03:09:02.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.146+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787745658}) 2024-09-16T03:09:02.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.146+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.146+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956706933}) 2024-09-16T03:09:02.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.146+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3409996872}) 2024-09-16T03:09:02.155 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946181217}) 2024-09-16T03:09:02.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1268874837}) 2024-09-16T03:09:02.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946181217}) 2024-09-16T03:09:02.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199192604}) 2024-09-16T03:09:02.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702955059}) 2024-09-16T03:09:02.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1360704327}) 2024-09-16T03:09:02.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702955059}) 2024-09-16T03:09:02.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.150+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=687388473}) 2024-09-16T03:09:02.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.154+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.154+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1427230150}) 2024-09-16T03:09:02.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.154+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274751973}) 2024-09-16T03:09:02.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.154+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.154+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1427230150}) 2024-09-16T03:09:02.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.154+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899032981}) 2024-09-16T03:09:02.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.158+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.158+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763888410}) 2024-09-16T03:09:02.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.158+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3492891030}) 2024-09-16T03:09:02.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.158+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.158+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763888410}) 2024-09-16T03:09:02.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.162+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3172789896}) 2024-09-16T03:09:02.168 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.162+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.168 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.162+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420506451}) 2024-09-16T03:09:02.168 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.162+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2493597530}) 2024-09-16T03:09:02.168 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.162+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.169 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.162+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420506451}) 2024-09-16T03:09:02.169 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.162+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3270181400}) 2024-09-16T03:09:02.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.166+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.166+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820049277}) 2024-09-16T03:09:02.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.166+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=211868364}) 2024-09-16T03:09:02.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.166+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.166+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820049277}) 2024-09-16T03:09:02.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.166+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1551173201}) 2024-09-16T03:09:02.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016726192}) 2024-09-16T03:09:02.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775097419}) 2024-09-16T03:09:02.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016726192}) 2024-09-16T03:09:02.176 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3640703254}) 2024-09-16T03:09:02.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180548797}) 2024-09-16T03:09:02.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1972001359}) 2024-09-16T03:09:02.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180548797}) 2024-09-16T03:09:02.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.170+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227206652}) 2024-09-16T03:09:02.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3228450860}) 2024-09-16T03:09:02.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1189556822}) 2024-09-16T03:09:02.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3228450860}) 2024-09-16T03:09:02.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1676135311}) 2024-09-16T03:09:02.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311492020}) 2024-09-16T03:09:02.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2663551089}) 2024-09-16T03:09:02.180 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.180 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311492020}) 2024-09-16T03:09:02.180 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964975129}) 2024-09-16T03:09:02.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=199586178}) 2024-09-16T03:09:02.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.174+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1372974584}) 2024-09-16T03:09:02.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=199586178}) 2024-09-16T03:09:02.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2790704851}) 2024-09-16T03:09:02.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1036823455}) 2024-09-16T03:09:02.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3856190934}) 2024-09-16T03:09:02.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1036823455}) 2024-09-16T03:09:02.183 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3763099501}) 2024-09-16T03:09:02.183 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.183 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230842305}) 2024-09-16T03:09:02.183 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3395042006}) 2024-09-16T03:09:02.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230842305}) 2024-09-16T03:09:02.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.178+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3962314788}) 2024-09-16T03:09:02.185 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.185 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445326993}) 2024-09-16T03:09:02.185 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=672996765}) 2024-09-16T03:09:02.185 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.185 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445326993}) 2024-09-16T03:09:02.185 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598383614}) 2024-09-16T03:09:02.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840008917}) 2024-09-16T03:09:02.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3088290439}) 2024-09-16T03:09:02.187 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.187 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840008917}) 2024-09-16T03:09:02.187 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892862400}) 2024-09-16T03:09:02.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199731355}) 2024-09-16T03:09:02.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4223995666}) 2024-09-16T03:09:02.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199731355}) 2024-09-16T03:09:02.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.182+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3015718487}) 2024-09-16T03:09:02.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400505701}) 2024-09-16T03:09:02.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1093174351}) 2024-09-16T03:09:02.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400505701}) 2024-09-16T03:09:02.190 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737352816}) 2024-09-16T03:09:02.190 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.190 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952633869}) 2024-09-16T03:09:02.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1694604305}) 2024-09-16T03:09:02.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952633869}) 2024-09-16T03:09:02.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2579105596}) 2024-09-16T03:09:02.192 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.192 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283160986}) 2024-09-16T03:09:02.192 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4171146100}) 2024-09-16T03:09:02.192 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.192 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283160986}) 2024-09-16T03:09:02.192 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.186+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2283789433}) 2024-09-16T03:09:02.194 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.194 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=2,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=24,osdmap=20} crc {logm=4076802760,mdsmap=2287870078,mgr=1407849713,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=617409152,mon_config_key=2135066834,monmap=3372108098,nvmeofgw=598337686,osd_metadata=266140046,osd_pg_creating=1610168463,osd_snap=3020911709,osdmap=489458172}) 2024-09-16T03:09:02.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=19,monmap=2,nvmeofgw=3,osd_metadata=5,osd_snap=24,osdmap=21} crc {logm=771841366,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1606013778,mon_config_key=2340868515,monmap=127340147,nvmeofgw=598337686,osd_metadata=3649966885,osd_snap=1765569357,osdmap=1879436468}) 2024-09-16T03:09:02.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=2,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=24,osdmap=20} crc {logm=4076802760,mdsmap=2287870078,mgr=1407849713,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=617409152,mon_config_key=2135066834,monmap=3372108098,nvmeofgw=598337686,osd_metadata=266140046,osd_pg_creating=1610168463,osd_snap=3020911709,osdmap=489458172}) 2024-09-16T03:09:02.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=24,osdmap=23} crc {logm=771841366,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=4119296792,mon_config_key=2033409059,monmap=3372108098,nvmeofgw=1596038039,osd_metadata=2073814740,osd_pg_creating=1610168463,osd_snap=175395951,osdmap=3622331430}) 2024-09-16T03:09:02.196 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.196 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3693472133}) 2024-09-16T03:09:02.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2559032982}) 2024-09-16T03:09:02.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3693472133}) 2024-09-16T03:09:02.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.190+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1653698188}) 2024-09-16T03:09:02.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.194+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.194+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3370428594}) 2024-09-16T03:09:02.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.194+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=334060702}) 2024-09-16T03:09:02.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.194+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.194+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3370428594}) 2024-09-16T03:09:02.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.194+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3200939992}) 2024-09-16T03:09:02.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.202+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.202+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=983139768}) 2024-09-16T03:09:02.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.202+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=57} crc {osdmap=3006161553}) 2024-09-16T03:09:02.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.202+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:09:02.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.202+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=983139768}) 2024-09-16T03:09:02.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:09:02.202+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=208135523}) 2024-09-16T03:09:03.153 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:09:03.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:09:03.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient: get_monmap_and_config 2024-09-16T03:09:03.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient: build_initial_monmap 2024-09-16T03:09:03.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient: monmap: 2024-09-16T03:09:03.286 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:03.286 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.287 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:03.285482+0000 2024-09-16T03:09:03.287 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:03.285482+0000 2024-09-16T03:09:03.287 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:03.287 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.287 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:03.287 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:03.287 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:03.287 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 1 Processor -- start 2024-09-16T03:09:03.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 1 -- start start 2024-09-16T03:09:03.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient: init 2024-09-16T03:09:03.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:03.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:03.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient(hunting): picked mon.noname-c con 0x7f5d14137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:03.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient(hunting): picked mon.noname-a con 0x7f5d14138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1413e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d140a4f90 con 0x7f5d14138100 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d140a4cb0 con 0x7f5d14137d20 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1409dc70 con 0x7f5d1413e2a0 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient(hunting): authenticate will time out at 1233.934780s 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d0bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5d14138100 0x7f5d1413cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35446/0 (socket says 172.21.15.45:35446) 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d18a98640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5d1413e2a0 0x7f5d14140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55150/0 (socket says 172.21.15.45:55150) 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d0bfff640 1 -- 172.21.15.45:0/3143286637 learned_addr learned my addr 172.21.15.45:0/3143286637 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3804132976 0 0) 0x7f5d1409dc70 con 0x7f5d1413e2a0 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient(hunting): my global_id is 28877 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d00003530 con 0x7f5d1413e2a0 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2394728732 0 0) 0x7f5d140a4f90 con 0x7f5d14138100 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient(hunting): my global_id is 104136 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d1409dc70 con 0x7f5d14138100 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 192150437 0 0) 0x7f5d1409dc70 con 0x7f5d14138100 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 >> v1:172.21.15.45:6790/0 conn(0x7f5d1413e2a0 legacy=0x7f5d14140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 >> v1:172.21.15.46:6789/0 conn(0x7f5d14137d20 legacy=0x7f5d140a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: _finish_hunting 0 2024-09-16T03:09:03.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 monclient: found mon.noname-a 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1409cc50 con 0x7f5d14138100 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: _finish_auth 0 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: _check_auth_tickets 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 5 monclient: authenticate success, global_id 104136 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5cfc001680 con 0x7f5d14138100 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: dump: 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.293 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5cfc001820 con 0x7f5d14138100 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 1 -- 172.21.15.45:0/3143286637 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5cfc001c30 con 0x7f5d14138100 2024-09-16T03:09:03.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d1a4fb640 4 set_mon_vals no callback set 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.282+0000 7f5d137fe640 10 monclient: dump: 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: shutdown 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/3143286637 >> v1:172.21.15.45:6789/0 conn(0x7f5d14138100 legacy=0x7f5d1413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/3143286637 shutdown_connections 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/3143286637 >> 172.21.15.45:0/3143286637 conn(0x7f5d140aa110 msgr2=0x7f5d140aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/3143286637 shutdown_connections 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/3143286637 wait complete. 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: build_initial_monmap 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: monmap: 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 librados: starting msgr at 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 librados: starting objecter 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 Processor -- start 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- start start 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 librados: setting wanted keys 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 librados: calling monclient init 2024-09-16T03:09:03.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: init 2024-09-16T03:09:03.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:03.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:09:03.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient(hunting): picked mon.noname-a con 0x7f5d14137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:03.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient(hunting): picked mon.noname-c con 0x7f5d14138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:03.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:03.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1413c070 con 0x7f5d14137d20 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d14139740 con 0x7f5d1413e2a0 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d140a1300 con 0x7f5d14138100 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient(hunting): authenticate will time out at 1233.937872s 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d13fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5d14137d20 0x7f5d14138920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35460/0 (socket says 172.21.15.45:35460) 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d0bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5d14138100 0x7f5d14139030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55162/0 (socket says 172.21.15.45:55162) 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d13fff640 1 -- 172.21.15.45:0/122159348 learned_addr learned my addr 172.21.15.45:0/122159348 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 client.?.objecter ms_handle_connect 0x7f5d14138100 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 client.?.objecter ms_handle_connect 0x7f5d14137d20 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3357245697 0 0) 0x7f5d1413c070 con 0x7f5d14137d20 2024-09-16T03:09:03.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): my global_id is 104139 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cf4004ec0 con 0x7f5d14137d20 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168920200 0 0) 0x7f5d140a1300 con 0x7f5d14138100 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): my global_id is 28880 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d1413c070 con 0x7f5d14138100 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 client.?.objecter ms_handle_connect 0x7f5d1413e2a0 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:03.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1273329587 0 0) 0x7f5cf4004ec0 con 0x7f5d14137d20 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d140a1300 con 0x7f5d14137d20 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5d04001a40 con 0x7f5d14137d20 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): dump: 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.304 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3150901240 0 0) 0x7f5d1413c070 con 0x7f5d14138100 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cf4004ec0 con 0x7f5d14138100 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5cfc001b70 con 0x7f5d14138100 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): dump: 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884608295 0 0) 0x7f5d14139740 con 0x7f5d1413e2a0 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient(hunting): my global_id is 22657 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d1413c070 con 0x7f5d1413e2a0 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 562352401 0 0) 0x7f5d140a1300 con 0x7f5d14137d20 2024-09-16T03:09:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 >> v1:172.21.15.45:6790/0 conn(0x7f5d14138100 legacy=0x7f5d14139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 >> v1:172.21.15.46:6789/0 conn(0x7f5d1413e2a0 legacy=0x7f5d141d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient: _finish_hunting 0 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 monclient: found mon.a 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1413b510 con 0x7f5d14137d20 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient: _finish_auth 0 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient: _check_auth_tickets 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 5 monclient: authenticate success, global_id 104139 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: _renew_subs 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d14139d10 con 0x7f5d14137d20 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 client.104139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: _renew_subs 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d1413a870 con 0x7f5d14137d20 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 librados: init done 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d04001d40 con 0x7f5d14137d20 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5d04003a70 con 0x7f5d14137d20 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d12ffd640 4 set_mon_vals no callback set 2024-09-16T03:09:03.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d117fa640 10 monclient: dump: 2024-09-16T03:09:03.309 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.309 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.309 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.309 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.309 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.309 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.309 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.286+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf8005090 con 0x7f5d14137d20 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.290+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f5d04003c10 con 0x7f5d14137d20 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.290+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(184..184 src has 1..184) ==== 5975+0+0 (unknown 1295916952 0 0) 0x7f5d0404ba60 con 0x7f5d14137d20 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.290+0000 7f5d117fa640 10 client.104139.objecter ms_dispatch 0x7f5d14000ba0 osd_map(184..184 src has 1..184) 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.290+0000 7f5d117fa640 3 client.104139.objecter handle_osd_map got epochs [184,184] > 0 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.290+0000 7f5d117fa640 3 client.104139.objecter handle_osd_map decoding full epoch 184 2024-09-16T03:09:03.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.290+0000 7f5d117fa640 20 client.104139.objecter dump_active .. 0 homeless 2024-09-16T03:09:03.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.290+0000 7f5d0bfff640 10 monclient: get_auth_request con 0x7f5cf403e2b0 auth_method 0 2024-09-16T03:09:03.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.290+0000 7f5d117fa640 10 client.104139.objecter ms_handle_connect 0x7f5cf403e2b0 2024-09-16T03:09:03.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.294+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5d04016b50 con 0x7f5d14137d20 2024-09-16T03:09:03.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.294+0000 7f5d117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:03.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.294+0000 7f5d117fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:03.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.530+0000 7f5d1a4fb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:09:03.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.530+0000 7f5d1a4fb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:03.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.530+0000 7f5d1a4fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.530+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5cf8005380 con 0x7f5d14137d20 2024-09-16T03:09:03.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.530+0000 7f5d117fa640 1 -- 172.21.15.45:0/122159348 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 3278863966) 0x7f5d04016450 con 0x7f5d14137d20 2024-09-16T03:09:03.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.530+0000 7f5d117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:03.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.530+0000 7f5d117fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:03.535 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:09:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.530+0000 7f5d1a4fb640 10 librados: watch_flush enter 2024-09-16T03:09:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 10 librados: watch_flush exit 2024-09-16T03:09:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 20 client.104139.objecter shutdown clearing up homeless session... 2024-09-16T03:09:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 10 client.104139.objecter successfully canceled tick 2024-09-16T03:09:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 >> v1:172.21.15.46:6812/149360280 conn(0x7f5cf403e2b0 legacy=0x7f5cf4040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 10 monclient: shutdown 2024-09-16T03:09:03.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 >> v1:172.21.15.45:6789/0 conn(0x7f5d14137d20 legacy=0x7f5d14138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 shutdown_connections 2024-09-16T03:09:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 >> 172.21.15.45:0/122159348 conn(0x7f5d140aa110 msgr2=0x7f5d140742f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 shutdown_connections 2024-09-16T03:09:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 1 -- 172.21.15.45:0/122159348 wait complete. 2024-09-16T03:09:03.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.534+0000 7f5d1a4fb640 1 librados: shutdown 2024-09-16T03:09:03.556 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:09:03.556 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:09:03.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:09:03.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: get_monmap_and_config 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: build_initial_monmap 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: monmap: 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:03.672702+0000 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:03.672702+0000 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:03.675 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 Processor -- start 2024-09-16T03:09:03.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 -- start start 2024-09-16T03:09:03.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: init 2024-09-16T03:09:03.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:03.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:09:03.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient(hunting): picked mon.noname-b con 0x7f32ec135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:03.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient(hunting): picked mon.noname-a con 0x7f32ec09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:03.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient(hunting): picked mon.noname-c con 0x7f32ec13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:03.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ec09f7e0 con 0x7f32ec09fc40 2024-09-16T03:09:03.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ec09b160 con 0x7f32ec13c010 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ec09b440 con 0x7f32ec135950 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient(hunting): authenticate will time out at 1234.321674s 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32eaffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f32ec135950 0x7f32ec135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55176/0 (socket says 172.21.15.45:55176) 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32eaffd640 1 -- 172.21.15.45:0/2577859301 learned_addr learned my addr 172.21.15.45:0/2577859301 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32ea7fc640 1 --1- 172.21.15.45:0/2577859301 >> v1:172.21.15.45:6789/0 conn(0x7f32ec09fc40 0x7f32ec13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35470/0 (socket says 172.21.15.45:35470) 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 647394257 0 0) 0x7f32ec09b440 con 0x7f32ec135950 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient(hunting): my global_id is 28883 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32d4003530 con 0x7f32ec135950 2024-09-16T03:09:03.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1980990306 0 0) 0x7f32ec09f7e0 con 0x7f32ec09fc40 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient(hunting): my global_id is 104142 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32ec09b440 con 0x7f32ec09fc40 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2243614150 0 0) 0x7f32ec09b440 con 0x7f32ec09fc40 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 >> v1:172.21.15.45:6790/0 conn(0x7f32ec135950 legacy=0x7f32ec135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 >> v1:172.21.15.46:6789/0 conn(0x7f32ec13c010 legacy=0x7f32ec13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 monclient: found mon.noname-a 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ec136440 con 0x7f32ec09fc40 2024-09-16T03:09:03.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 5 monclient: authenticate success, global_id 104142 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f32e0001680 con 0x7f32ec09fc40 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: dump: 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f32e0001820 con 0x7f32ec09fc40 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 4 set_mon_vals no callback set 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 1 -- 172.21.15.45:0/2577859301 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f32e0001c70 con 0x7f32ec09fc40 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32e9ffb640 10 monclient: dump: 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.686 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: shutdown 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 -- 172.21.15.45:0/2577859301 >> v1:172.21.15.45:6789/0 conn(0x7f32ec09fc40 legacy=0x7f32ec13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 -- 172.21.15.45:0/2577859301 shutdown_connections 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 -- 172.21.15.45:0/2577859301 >> 172.21.15.45:0/2577859301 conn(0x7f32ec0a2bd0 msgr2=0x7f32ec0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 -- 172.21.15.45:0/2577859301 shutdown_connections 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 1 -- 172.21.15.45:0/2577859301 wait complete. 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: build_initial_monmap 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.670+0000 7f32f14d6640 10 monclient: monmap: 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 librados: starting msgr at 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 librados: starting objecter 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 Processor -- start 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 -- start start 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 librados: setting wanted keys 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 librados: calling monclient init 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient: init 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient(hunting): picked mon.noname-b con 0x7f32ec09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient(hunting): picked mon.noname-a con 0x7f32ec135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient(hunting): picked mon.noname-c con 0x7f32ec13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ec0718b0 con 0x7f32ec135950 2024-09-16T03:09:03.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ec06f1d0 con 0x7f32ec09fc40 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ec09c460 con 0x7f32ec13c010 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient(hunting): authenticate will time out at 1234.324878s 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32ea7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f32ec135950 0x7f32ec06eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35486/0 (socket says 172.21.15.45:35486) 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32ea7fc640 1 -- 172.21.15.45:0/2945248724 learned_addr learned my addr 172.21.15.45:0/2945248724 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.?.objecter ms_handle_connect 0x7f32ec13c010 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.?.objecter ms_handle_connect 0x7f32ec135950 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2743467358 0 0) 0x7f32ec09c460 con 0x7f32ec13c010 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): my global_id is 28886 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32d8004ec0 con 0x7f32ec13c010 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057654082 0 0) 0x7f32ec0718b0 con 0x7f32ec135950 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): my global_id is 104145 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32ec09c460 con 0x7f32ec135950 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.?.objecter ms_handle_connect 0x7f32ec09fc40 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 911292544 0 0) 0x7f32ec09c460 con 0x7f32ec135950 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32ec0718b0 con 0x7f32ec135950 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f32e0001c70 con 0x7f32ec135950 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): dump: 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:03.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1144372541 0 0) 0x7f32d8004ec0 con 0x7f32ec13c010 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32ec09c460 con 0x7f32ec13c010 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 157031083 0 0) 0x7f32ec06f1d0 con 0x7f32ec09fc40 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): my global_id is 22663 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32d8004ec0 con 0x7f32ec09fc40 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f32e4001800 con 0x7f32ec13c010 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): dump: 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.693 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3213075001 0 0) 0x7f32ec0718b0 con 0x7f32ec135950 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 >> v1:172.21.15.45:6790/0 conn(0x7f32ec13c010 legacy=0x7f32ec1d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 >> v1:172.21.15.46:6789/0 conn(0x7f32ec09fc40 legacy=0x7f32ec06e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 monclient: found mon.a 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:03.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ec074150 con 0x7f32ec135950 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: _finish_auth 0 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 5 monclient: authenticate success, global_id 104145 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient: _renew_subs 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f32ec0715d0 con 0x7f32ec135950 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 client.104145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient: _renew_subs 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f32ec13b1a0 con 0x7f32ec135950 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 librados: init done 2024-09-16T03:09:03.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f32e00017c0 con 0x7f32ec135950 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f32e0004c40 con 0x7f32ec135950 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32e97fa640 4 set_mon_vals no callback set 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: dump: 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.696 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f32cc0065d0 con 0x7f32cc002870 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.104145.objecter ms_handle_connect 0x7f32cc002870 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.104145.objecter resend_mon_ops 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f32e0010cb0 con 0x7f32ec135950 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(184..184 src has 1..184) ==== 5975+0+0 (unknown 1295916952 0 0) 0x7f32e004b920 con 0x7f32ec135950 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.104145.objecter ms_dispatch 0x7f32ec000ba0 osd_map(184..184 src has 1..184) 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 3 client.104145.objecter handle_osd_map got epochs [184,184] > 0 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 3 client.104145.objecter handle_osd_map decoding full epoch 184 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 20 client.104145.objecter dump_active .. 0 homeless 2024-09-16T03:09:03.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3498591047 0 0) 0x7f32cc0065d0 con 0x7f32cc002870 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient(hunting): my global_id is 104148 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32ec0718b0 con 0x7f32cc002870 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 524397674 0 0) 0x7f32ec0718b0 con 0x7f32cc002870 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32cc0065d0 con 0x7f32cc002870 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f32dc001550 con 0x7f32cc002870 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32eb7fe640 10 monclient: get_auth_request con 0x7f32d803e240 auth_method 0 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: dump: 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.698 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2008956422 0 0) 0x7f32cc0065d0 con 0x7f32cc002870 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32cc00a170 con 0x7f32cc002870 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 client.104145.objecter ms_handle_connect 0x7f32d803e240 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f32dc004b70 con 0x7f32cc002870 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:03.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.674+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 >> v1:172.21.15.45:6789/0 conn(0x7f32cc002870 legacy=0x7f32cc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.678+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 >> v1:172.21.15.45:6789/0 conn(0x7f32cc002870 legacy=0x7f32cc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:09:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f32cc009e80 con 0x7f32cc009140 2024-09-16T03:09:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 client.104145.objecter ms_handle_connect 0x7f32cc009140 2024-09-16T03:09:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 client.104145.objecter resend_mon_ops 2024-09-16T03:09:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 32591881 0 0) 0x7f32cc009e80 con 0x7f32cc009140 2024-09-16T03:09:03.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 monclient(hunting): my global_id is 104151 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32cc0065d0 con 0x7f32cc009140 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 179474725 0 0) 0x7f32cc0065d0 con 0x7f32cc009140 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32cc009e80 con 0x7f32cc009140 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f32dc004e00 con 0x7f32cc009140 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 monclient: dump: 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2953988117 0 0) 0x7f32cc009e80 con 0x7f32cc009140 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f32cc00c400 con 0x7f32cc009140 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3968 (unknown 0 0 3184023593) 0x7f32dc005f90 con 0x7f32cc009140 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 >> v1:172.21.15.45:6789/0 conn(0x7f32cc009140 legacy=0x7f32cc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32d37fe640 1 -- 172.21.15.45:0/2945248724 >> v1:172.21.15.45:6789/0 conn(0x7f32cc009140 legacy=0x7f32cc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 20, 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 2783, 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:09:03.704 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:03.705 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:03.706 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:09:03.707 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:03.708 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:03.709 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:03.710 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.711 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:09:03.712 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.713 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:03.714 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:03.715 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:09:03.715 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.715 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.715 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:09:03.715 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 10 librados: watch_flush enter 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 10 librados: watch_flush exit 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 20 client.104145.objecter shutdown clearing up homeless session... 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 10 client.104145.objecter successfully canceled tick 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 >> v1:172.21.15.46:6812/149360280 conn(0x7f32d803e240 legacy=0x7f32d8040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 10 monclient: shutdown 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 >> v1:172.21.15.45:6789/0 conn(0x7f32ec135950 legacy=0x7f32ec06eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32eb7fe640 1 -- 172.21.15.45:0/2945248724 reap_dead start 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32eb7fe640 1 -- 172.21.15.45:0/2945248724 reap_dead start 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 shutdown_connections 2024-09-16T03:09:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 >> 172.21.15.45:0/2945248724 conn(0x7f32ec0a2bd0 msgr2=0x7f32ec13ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:03.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 shutdown_connections 2024-09-16T03:09:03.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 1 -- 172.21.15.45:0/2945248724 wait complete. 2024-09-16T03:09:03.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.682+0000 7f32f14d6640 1 librados: shutdown 2024-09-16T03:09:03.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:09:03.718 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:09:03.718 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:09:03.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient: get_monmap_and_config 2024-09-16T03:09:03.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient: build_initial_monmap 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient: monmap: 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:03.833707+0000 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:03.833707+0000 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:03.835 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 1 Processor -- start 2024-09-16T03:09:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 1 -- start start 2024-09-16T03:09:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient: init 2024-09-16T03:09:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec809c570 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec8134320 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec81347d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec8135150 con 0x7f4ec81347d0 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec8074470 con 0x7f4ec809c570 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec8074750 con 0x7f4ec8134320 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 10 monclient(hunting): authenticate will time out at 1234.483031s 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecc964640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4ec8134320 0x7f4ec813cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55202/0 (socket says 172.21.15.45:55202) 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecc964640 1 -- 172.21.15.45:0/3398122777 learned_addr learned my addr 172.21.15.45:0/3398122777 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecd966640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4ec81347d0 0x7f4ec81402a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35496/0 (socket says 172.21.15.45:35496) 2024-09-16T03:09:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2406692244 0 0) 0x7f4ec4001120 con 0x7f4ec81347d0 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient(hunting): my global_id is 104154 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4eb4003530 con 0x7f4ec81347d0 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628509657 0 0) 0x7f4ec8074750 con 0x7f4ec8134320 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient(hunting): my global_id is 28889 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec4001120 con 0x7f4ec8134320 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3646634189 0 0) 0x7f4eb4003530 con 0x7f4ec81347d0 2024-09-16T03:09:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 >> v1:172.21.15.45:6790/0 conn(0x7f4ec8134320 legacy=0x7f4ec813cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 >> v1:172.21.15.46:6789/0 conn(0x7f4ec809c570 legacy=0x7f4ec809c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient: _finish_hunting 0 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 monclient: found mon.noname-a 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec8074a30 con 0x7f4ec81347d0 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient: _finish_auth 0 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient: _check_auth_tickets 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4ec4001870 con 0x7f4ec81347d0 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ebffff640 10 monclient: dump: 2024-09-16T03:09:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 5 monclient: authenticate success, global_id 104154 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.830+0000 7f4ecf3c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ec4001a10 con 0x7f4ec81347d0 2024-09-16T03:09:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebffff640 1 -- 172.21.15.45:0/3398122777 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4ec40047f0 con 0x7f4ec81347d0 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 4 set_mon_vals no callback set 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebffff640 10 monclient: dump: 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: shutdown 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3398122777 >> v1:172.21.15.45:6789/0 conn(0x7f4ec81347d0 legacy=0x7f4ec81402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3398122777 shutdown_connections 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3398122777 >> 172.21.15.45:0/3398122777 conn(0x7f4ec80a2bd0 msgr2=0x7f4ec80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3398122777 shutdown_connections 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3398122777 wait complete. 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: build_initial_monmap 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: monmap: 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 librados: starting msgr at 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 librados: starting objecter 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 Processor -- start 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- start start 2024-09-16T03:09:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 librados: setting wanted keys 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 librados: calling monclient init 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: init 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec809c570 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec8134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec81347d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec81cdbd0 con 0x7f4ec8134320 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec809c030 con 0x7f4ec81347d0 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec81332d0 con 0x7f4ec809c570 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient(hunting): authenticate will time out at 1234.486112s 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecd165640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4ec809c570 0x7f4ec81ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55210/0 (socket says 172.21.15.45:55210) 2024-09-16T03:09:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecd165640 1 -- 172.21.15.45:0/3138992960 learned_addr learned my addr 172.21.15.45:0/3138992960 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 client.?.objecter ms_handle_connect 0x7f4ec809c570 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 client.?.objecter ms_handle_connect 0x7f4ec8134320 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1533002594 0 0) 0x7f4ec81332d0 con 0x7f4ec809c570 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): my global_id is 28892 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4eac004ec0 con 0x7f4ec809c570 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 976097002 0 0) 0x7f4ec81cdbd0 con 0x7f4ec8134320 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): my global_id is 104157 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec81332d0 con 0x7f4ec8134320 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 client.?.objecter ms_handle_connect 0x7f4ec81347d0 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1150398794 0 0) 0x7f4eac004ec0 con 0x7f4ec809c570 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ec81cdbd0 con 0x7f4ec809c570 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4ec0001dc0 con 0x7f4ec809c570 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): dump: 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 679864109 0 0) 0x7f4ec81332d0 con 0x7f4ec8134320 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4eac004ec0 con 0x7f4ec8134320 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4eb80012e0 con 0x7f4ec8134320 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): dump: 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443726598 0 0) 0x7f4ec809c030 con 0x7f4ec81347d0 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient(hunting): my global_id is 22669 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec81332d0 con 0x7f4ec81347d0 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2622924205 0 0) 0x7f4ec81cdbd0 con 0x7f4ec809c570 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 >> v1:172.21.15.46:6789/0 conn(0x7f4ec81347d0 legacy=0x7f4ec81d5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 >> v1:172.21.15.45:6789/0 conn(0x7f4ec8134320 legacy=0x7f4ec81cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient: _finish_hunting 0 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 monclient: found mon.c 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec8132620 con 0x7f4ec809c570 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient: _finish_auth 0 2024-09-16T03:09:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient: _check_auth_tickets 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 5 monclient: authenticate success, global_id 28892 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: _renew_subs 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec8099aa0 con 0x7f4ec809c570 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 client.28892.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: _renew_subs 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ec8131150 con 0x7f4ec809c570 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 1 librados: init done 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ec00020c0 con 0x7f4ec809c570 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4ec0003df0 con 0x7f4ec809c570 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebf7fe640 4 set_mon_vals no callback set 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ebdffb640 10 monclient: dump: 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.834+0000 7f4ecf3c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4ea80065d0 con 0x7f4ea8002870 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 client.28892.objecter ms_handle_connect 0x7f4ea8002870 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 client.28892.objecter resend_mon_ops 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2228905896 0 0) 0x7f4ea80065d0 con 0x7f4ea8002870 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 monclient(hunting): my global_id is 28895 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec81cdbd0 con 0x7f4ea8002870 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4ec0003f90 con 0x7f4ec809c570 2024-09-16T03:09:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(184..184 src has 1..184) ==== 5975+0+0 (unknown 1295916952 0 0) 0x7f4ec004be10 con 0x7f4ec809c570 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 client.28892.objecter ms_dispatch 0x7f4ec8000ba0 osd_map(184..184 src has 1..184) 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 3 client.28892.objecter handle_osd_map got epochs [184,184] > 0 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 3 client.28892.objecter handle_osd_map decoding full epoch 184 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 20 client.28892.objecter dump_active .. 0 homeless 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3855203955 0 0) 0x7f4ec81cdbd0 con 0x7f4ea8002870 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ea80065d0 con 0x7f4ea8002870 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4eb8004b80 con 0x7f4ea8002870 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 monclient: dump: 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2860295994 0 0) 0x7f4ea80065d0 con 0x7f4ea8002870 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ea800a170 con 0x7f4ea8002870 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ecd966640 10 monclient: get_auth_request con 0x7f4eac03e220 auth_method 0 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4eb8004da0 con 0x7f4ea8002870 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:03.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 >> v1:172.21.15.45:6790/0 conn(0x7f4ea8002870 legacy=0x7f4ea8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 >> v1:172.21.15.45:6790/0 conn(0x7f4ea8002870 legacy=0x7f4ea8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:03.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.838+0000 7f4ebdffb640 10 client.28892.objecter ms_handle_connect 0x7f4eac03e220 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ecf3c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4ea8009e80 con 0x7f4ea8009140 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ebdffb640 10 client.28892.objecter ms_handle_connect 0x7f4ea8009140 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ebdffb640 10 client.28892.objecter resend_mon_ops 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235118545 0 0) 0x7f4ea8009e80 con 0x7f4ea8009140 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ebdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ebdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ebdffb640 10 monclient(hunting): my global_id is 28898 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.842+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ea80065d0 con 0x7f4ea8009140 2024-09-16T03:09:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2278435175 0 0) 0x7f4ea80065d0 con 0x7f4ea8009140 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ea8009e80 con 0x7f4ea8009140 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4eb8005480 con 0x7f4ea8009140 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 10 monclient: dump: 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3894533413 0 0) 0x7f4ea8009e80 con 0x7f4ea8009140 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ea800c400 con 0x7f4ea8009140 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 187207839) 0x7f4eb80055c0 con 0x7f4ea8009140 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 >> v1:172.21.15.45:6790/0 conn(0x7f4ea8009140 legacy=0x7f4ea80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ebdffb640 1 -- 172.21.15.45:0/3138992960 >> v1:172.21.15.45:6790/0 conn(0x7f4ea8009140 legacy=0x7f4ea80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:09:03.865 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 20, 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 265699, 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:03.866 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:03.867 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:09:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:09:03.869 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:03.870 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.871 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:03.872 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:03.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:09:03.874 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.875 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:09:03.876 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:09:03.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 10 librados: watch_flush enter 2024-09-16T03:09:03.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 10 librados: watch_flush exit 2024-09-16T03:09:03.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 20 client.28892.objecter shutdown clearing up homeless session... 2024-09-16T03:09:03.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 10 client.28892.objecter successfully canceled tick 2024-09-16T03:09:03.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 >> v1:172.21.15.46:6812/149360280 conn(0x7f4eac03e220 legacy=0x7f4eac0406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 10 monclient: shutdown 2024-09-16T03:09:03.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:03.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 >> v1:172.21.15.45:6790/0 conn(0x7f4ec809c570 legacy=0x7f4ec81ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecd966640 1 -- 172.21.15.45:0/3138992960 reap_dead start 2024-09-16T03:09:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecd966640 1 -- 172.21.15.45:0/3138992960 reap_dead start 2024-09-16T03:09:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 shutdown_connections 2024-09-16T03:09:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 >> 172.21.15.45:0/3138992960 conn(0x7f4ec80a2bd0 msgr2=0x7f4ec80a3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 shutdown_connections 2024-09-16T03:09:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 1 -- 172.21.15.45:0/3138992960 wait complete. 2024-09-16T03:09:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.846+0000 7f4ecf3c9640 1 librados: shutdown 2024-09-16T03:09:03.879 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:09:03.879 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:09:03.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient: get_monmap_and_config 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient: build_initial_monmap 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient: monmap: 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:03.997017+0000 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:03.997017+0000 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:03.998 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:03.999 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 1 Processor -- start 2024-09-16T03:09:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 1 -- start start 2024-09-16T03:09:03.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient: init 2024-09-16T03:09:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:04.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient(hunting): picked mon.noname-a con 0x7fdf58137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient(hunting): picked mon.noname-c con 0x7fdf58138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient(hunting): picked mon.noname-b con 0x7fdf5813e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf580a4f90 con 0x7fdf58137d20 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf580a4cb0 con 0x7fdf58138100 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf5809dc70 con 0x7fdf5813e2a0 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5eb38640 10 monclient(hunting): authenticate will time out at 1234.647185s 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5d0d5640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdf5813e2a0 0x7fdf58140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55246/0 (socket says 172.21.15.45:55246) 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5d0d5640 1 -- 172.21.15.45:0/2351685117 learned_addr learned my addr 172.21.15.45:0/2351685117 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf5c8d4640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdf58137d20 0x7fdf580a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35520/0 (socket says 172.21.15.45:35520) 2024-09-16T03:09:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2256520856 0 0) 0x7fdf580a4f90 con 0x7fdf58137d20 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 10 monclient(hunting): my global_id is 104160 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf40003530 con 0x7fdf58137d20 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 327264825 0 0) 0x7fdf5809dc70 con 0x7fdf5813e2a0 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 10 monclient(hunting): my global_id is 28901 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.994+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf580a4f90 con 0x7fdf5813e2a0 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104961281 0 0) 0x7fdf580a4cb0 con 0x7fdf58138100 2024-09-16T03:09:04.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient(hunting): my global_id is 22672 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf5809dc70 con 0x7fdf58138100 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3700190994 0 0) 0x7fdf40003530 con 0x7fdf58137d20 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 >> v1:172.21.15.45:6790/0 conn(0x7fdf5813e2a0 legacy=0x7fdf58140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 >> v1:172.21.15.46:6789/0 conn(0x7fdf58138100 legacy=0x7fdf5813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 monclient: found mon.noname-a 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5809cc50 con 0x7fdf58137d20 2024-09-16T03:09:04.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdf540015d0 con 0x7fdf58137d20 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 5 monclient: authenticate success, global_id 104160 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: dump: 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf540017f0 con 0x7fdf58137d20 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 1 -- 172.21.15.45:0/2351685117 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdf54001c40 con 0x7fdf58137d20 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 4 set_mon_vals no callback set 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4f7fe640 10 monclient: dump: 2024-09-16T03:09:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: shutdown 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2351685117 >> v1:172.21.15.45:6789/0 conn(0x7fdf58137d20 legacy=0x7fdf580a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2351685117 shutdown_connections 2024-09-16T03:09:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2351685117 >> 172.21.15.45:0/2351685117 conn(0x7fdf580aa110 msgr2=0x7fdf580aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2351685117 shutdown_connections 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2351685117 wait complete. 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: build_initial_monmap 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: monmap: 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:04.009 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 librados: starting msgr at 2024-09-16T03:09:04.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 librados: starting objecter 2024-09-16T03:09:04.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 Processor -- start 2024-09-16T03:09:04.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- start start 2024-09-16T03:09:04.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 librados: setting wanted keys 2024-09-16T03:09:04.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 librados: calling monclient init 2024-09-16T03:09:04.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: init 2024-09-16T03:09:04.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:04.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:04.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient(hunting): picked mon.noname-b con 0x7fdf58137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:04.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient(hunting): picked mon.noname-a con 0x7fdf58138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:04.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient(hunting): picked mon.noname-c con 0x7fdf5813e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:04.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf580a1300 con 0x7fdf58138100 2024-09-16T03:09:04.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf5813b970 con 0x7fdf58137d20 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf5813bc50 con 0x7fdf5813e2a0 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient(hunting): authenticate will time out at 1234.650637s 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5d0d5640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdf5813e2a0 0x7fdf581d5140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55260/0 (socket says 172.21.15.45:55260) 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdf58138100 0x7fdf58139030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35530/0 (socket says 172.21.15.45:35530) 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5d0d5640 1 -- 172.21.15.45:0/2717068141 learned_addr learned my addr 172.21.15.45:0/2717068141 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 client.?.objecter ms_handle_connect 0x7fdf5813e2a0 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 client.?.objecter ms_handle_connect 0x7fdf58138100 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538448496 0 0) 0x7fdf5813bc50 con 0x7fdf5813e2a0 2024-09-16T03:09:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): my global_id is 28904 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf44004ec0 con 0x7fdf5813e2a0 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395649101 0 0) 0x7fdf580a1300 con 0x7fdf58138100 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): my global_id is 104163 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf5813bc50 con 0x7fdf58138100 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 client.?.objecter ms_handle_connect 0x7fdf58137d20 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1448553083 0 0) 0x7fdf44004ec0 con 0x7fdf5813e2a0 2024-09-16T03:09:04.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf580a1300 con 0x7fdf5813e2a0 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdf50001c80 con 0x7fdf5813e2a0 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): dump: 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1478657631 0 0) 0x7fdf5813b970 con 0x7fdf58137d20 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): my global_id is 22675 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf44004ec0 con 0x7fdf58137d20 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 925220887 0 0) 0x7fdf5813bc50 con 0x7fdf58138100 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf5813b970 con 0x7fdf58138100 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdf480018c0 con 0x7fdf58138100 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:04.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient(hunting): dump: 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 267430820 0 0) 0x7fdf580a1300 con 0x7fdf5813e2a0 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 >> v1:172.21.15.46:6789/0 conn(0x7fdf58137d20 legacy=0x7fdf58138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 >> v1:172.21.15.45:6789/0 conn(0x7fdf58138100 legacy=0x7fdf58139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient: _finish_hunting 0 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 monclient: found mon.c 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5813d910 con 0x7fdf5813e2a0 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 5 monclient: authenticate success, global_id 28904 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: _renew_subs 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5813b640 con 0x7fdf5813e2a0 2024-09-16T03:09:04.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 client.28904.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: _renew_subs 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf58139740 con 0x7fdf5813e2a0 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf5eb38640 1 librados: init done 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf50004b70 con 0x7fdf5813e2a0 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4effd640 4 set_mon_vals no callback set 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdf50004ec0 con 0x7fdf5813e2a0 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:03.998+0000 7fdf4d7fa640 10 monclient: dump: 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.023 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf5eb38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdf380065d0 con 0x7fdf38002870 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fdf50010fb0 con 0x7fdf5813e2a0 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(184..184 src has 1..184) ==== 5975+0+0 (unknown 1295916952 0 0) 0x7fdf5004be10 con 0x7fdf5813e2a0 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 client.28904.objecter ms_dispatch 0x7fdf58000ba0 osd_map(184..184 src has 1..184) 2024-09-16T03:09:04.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 3 client.28904.objecter handle_osd_map got epochs [184,184] > 0 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 3 client.28904.objecter handle_osd_map decoding full epoch 184 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 20 client.28904.objecter dump_active .. 0 homeless 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 client.28904.objecter ms_handle_connect 0x7fdf38002870 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 client.28904.objecter resend_mon_ops 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4ffff640 10 monclient: get_auth_request con 0x7fdf4403e260 auth_method 0 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 688797273 0 0) 0x7fdf380065d0 con 0x7fdf38002870 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 monclient(hunting): my global_id is 22678 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf580a1300 con 0x7fdf38002870 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 client.28904.objecter ms_handle_connect 0x7fdf4403e260 2024-09-16T03:09:04.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 486067473 0 0) 0x7fdf580a1300 con 0x7fdf38002870 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf380065d0 con 0x7fdf38002870 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdf54001880 con 0x7fdf38002870 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 monclient: dump: 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.026 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 940813478 0 0) 0x7fdf380065d0 con 0x7fdf38002870 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf3800a170 con 0x7fdf38002870 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdf54001950 con 0x7fdf38002870 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 >> v1:172.21.15.46:6789/0 conn(0x7fdf38002870 legacy=0x7fdf38004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.002+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 >> v1:172.21.15.46:6789/0 conn(0x7fdf38002870 legacy=0x7fdf38004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf5eb38640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdf38009e80 con 0x7fdf38009140 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 10 client.28904.objecter ms_handle_connect 0x7fdf38009140 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 10 client.28904.objecter resend_mon_ops 2024-09-16T03:09:04.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2824655429 0 0) 0x7fdf38009e80 con 0x7fdf38009140 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 10 monclient(hunting): my global_id is 22681 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf380065d0 con 0x7fdf38009140 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3178515007 0 0) 0x7fdf380065d0 con 0x7fdf38009140 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf38009e80 con 0x7fdf38009140 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdf54004a30 con 0x7fdf38009140 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 10 monclient: dump: 2024-09-16T03:09:04.028 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2882587572 0 0) 0x7fdf38009e80 con 0x7fdf38009140 2024-09-16T03:09:04.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.010+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf3800c420 con 0x7fdf38009140 2024-09-16T03:09:04.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 1442419976) 0x7fdf54005e60 con 0x7fdf38009140 2024-09-16T03:09:04.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf4d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:09:04.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:04.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 >> v1:172.21.15.46:6789/0 conn(0x7fdf38009140 legacy=0x7fdf380095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf4d7fa640 1 -- 172.21.15.45:0/2717068141 >> v1:172.21.15.46:6789/0 conn(0x7fdf38009140 legacy=0x7fdf380095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:04.030 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 20, 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 265816, 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:04.031 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:04.032 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:04.033 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:09:04.034 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:09:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:09:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:09:04.043 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:09:04.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 10 librados: watch_flush enter 2024-09-16T03:09:04.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 10 librados: watch_flush exit 2024-09-16T03:09:04.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 20 client.28904.objecter shutdown clearing up homeless session... 2024-09-16T03:09:04.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 10 client.28904.objecter successfully canceled tick 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 >> v1:172.21.15.46:6812/149360280 conn(0x7fdf4403e260 legacy=0x7fdf44040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 10 monclient: shutdown 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 >> v1:172.21.15.45:6790/0 conn(0x7fdf5813e2a0 legacy=0x7fdf581d5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5d0d5640 1 -- 172.21.15.45:0/2717068141 reap_dead start 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5d0d5640 1 -- 172.21.15.45:0/2717068141 reap_dead start 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 shutdown_connections 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 >> 172.21.15.45:0/2717068141 conn(0x7fdf580aa110 msgr2=0x7fdf580a09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 shutdown_connections 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 1 -- 172.21.15.45:0/2717068141 wait complete. 2024-09-16T03:09:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.014+0000 7fdf5eb38640 1 librados: shutdown 2024-09-16T03:09:04.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:09:04.046 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:09:04.046 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:09:04.046 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:09:04.046 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:09:04.046 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:09:04.046 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:09:04.060 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:09:04.061 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:09:04.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:09:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient: get_monmap_and_config 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient: build_initial_monmap 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient: monmap: 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:04.217914+0000 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:04.217914+0000 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:04.219 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:04.220 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 1 Processor -- start 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 1 -- start start 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient: init 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient(hunting): picked mon.noname-a con 0x7f358409b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient(hunting): picked mon.noname-b con 0x7f358409b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient(hunting): picked mon.noname-c con 0x7f358413b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358409cc20 con 0x7f358409b8e0 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358409d820 con 0x7f358413b000 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358409b6a0 con 0x7f358409b250 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:04.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f358ac54640 10 monclient(hunting): authenticate will time out at 1234.867244s 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f3583fff640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f358409b250 legacy=0x7f35841398e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f35889f0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f358409b8e0 0x7f358409bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35556/0 (socket says 172.21.15.45:35556) 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f35889f0640 1 -- 172.21.15.45:0/3591879329 learned_addr learned my addr 172.21.15.45:0/3591879329 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f35837fe640 1 -- 172.21.15.45:0/3591879329 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2033654946 0 0) 0x7f358409cc20 con 0x7f358409b8e0 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f35837fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f35837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f35837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f35837fe640 10 monclient(hunting): my global_id is 104166 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.214+0000 7f35837fe640 1 -- 172.21.15.45:0/3591879329 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3564003530 con 0x7f358409b8e0 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 1 -- 172.21.15.45:0/3591879329 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4113560527 0 0) 0x7f3564003530 con 0x7f358409b8e0 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 1 -- 172.21.15.45:0/3591879329 >> v1:172.21.15.45:6790/0 conn(0x7f358409b250 legacy=0x7f35841398e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:09:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 1 -- 172.21.15.45:0/3591879329 >> v1:172.21.15.46:6789/0 conn(0x7f358413b000 legacy=0x7f358413d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: _finish_hunting 0 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 1 monclient: found mon.noname-a 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 1 -- 172.21.15.45:0/3591879329 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35840a21f0 con 0x7f358409b8e0 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: _finish_auth 0 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: _check_auth_tickets 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 1 -- 172.21.15.45:0/3591879329 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f35740016a0 con 0x7f358409b8e0 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 5 monclient: authenticate success, global_id 104166 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: dump: 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 1 -- 172.21.15.45:0/3591879329 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35740018c0 con 0x7f358409b8e0 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 4 set_mon_vals no callback set 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 1 -- 172.21.15.45:0/3591879329 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3574001cd0 con 0x7f358409b8e0 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35837fe640 10 monclient: dump: 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.227 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: shutdown 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- 172.21.15.45:0/3591879329 >> v1:172.21.15.45:6789/0 conn(0x7f358409b8e0 legacy=0x7f358409bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- 172.21.15.45:0/3591879329 shutdown_connections 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- 172.21.15.45:0/3591879329 >> 172.21.15.45:0/3591879329 conn(0x7f35840a2bd0 msgr2=0x7f35840a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- 172.21.15.45:0/3591879329 shutdown_connections 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- 172.21.15.45:0/3591879329 wait complete. 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: build_initial_monmap 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: monmap: 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 librados: starting msgr at 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 librados: starting objecter 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 Processor -- start 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- start start 2024-09-16T03:09:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 librados: setting wanted keys 2024-09-16T03:09:04.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 librados: calling monclient init 2024-09-16T03:09:04.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: init 2024-09-16T03:09:04.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:04.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient(hunting): picked mon.noname-a con 0x7f358409b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient(hunting): picked mon.noname-c con 0x7f358409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient(hunting): picked mon.noname-b con 0x7f358413b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3584092540 con 0x7f358409b250 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3584090a80 con 0x7f358413b000 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358413a4d0 con 0x7f358409b8e0 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient(hunting): authenticate will time out at 1234.870653s 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f3583fff640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f358409b8e0 legacy=0x7f35841cd500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35889f0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f358409b250 0x7f3584099360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35560/0 (socket says 172.21.15.45:35560) 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35889f0640 1 -- 172.21.15.45:0/1304512331 learned_addr learned my addr 172.21.15.45:0/1304512331 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 client.?.objecter ms_handle_connect 0x7f358409b250 2024-09-16T03:09:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487138407 0 0) 0x7f35840a21f0 con 0x7f358409b250 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient(hunting): my global_id is 104169 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3568004ef0 con 0x7f358409b250 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 client.?.objecter ms_handle_connect 0x7f358413b000 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 938744987 0 0) 0x7f3568004ef0 con 0x7f358409b250 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35840a21f0 con 0x7f358409b250 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3574001c10 con 0x7f358409b250 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient(hunting): dump: 2024-09-16T03:09:04.234 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2082907144 0 0) 0x7f35840a21f0 con 0x7f358409b250 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 >> v1:172.21.15.45:6790/0 conn(0x7f358409b8e0 legacy=0x7f35841cd500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 >> v1:172.21.15.46:6789/0 conn(0x7f358413b000 legacy=0x7f35841d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient: _finish_hunting 0 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 monclient: found mon.a 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3584092de0 con 0x7f358409b250 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient: _finish_auth 0 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient: _check_auth_tickets 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 5 monclient: authenticate success, global_id 104169 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: _renew_subs 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35840915e0 con 0x7f358409b250 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 client.104169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: _renew_subs 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3584099c40 con 0x7f358409b250 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f358ac54640 1 librados: init done 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3574004c00 con 0x7f358409b250 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3574004db0 con 0x7f358409b250 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f35817fa640 10 monclient: dump: 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:04.237 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.218+0000 7f3582ffd640 4 set_mon_vals no callback set 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f358ac54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f358ac54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f358ac54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f355c005090 con 0x7f358409b250 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f3574010e20 con 0x7f358409b250 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(184..184 src has 1..184) ==== 5975+0+0 (unknown 1295916952 0 0) 0x7f357404ba50 con 0x7f358409b250 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f35817fa640 10 client.104169.objecter ms_dispatch 0x7f3584000ba0 osd_map(184..184 src has 1..184) 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f35817fa640 3 client.104169.objecter handle_osd_map got epochs [184,184] > 0 2024-09-16T03:09:04.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f35817fa640 3 client.104169.objecter handle_osd_map decoding full epoch 184 2024-09-16T03:09:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f35817fa640 20 client.104169.objecter dump_active .. 0 homeless 2024-09-16T03:09:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f3583fff640 10 monclient: get_auth_request con 0x7f356803dfc0 auth_method 0 2024-09-16T03:09:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.222+0000 7f35817fa640 10 client.104169.objecter ms_handle_connect 0x7f356803dfc0 2024-09-16T03:09:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.226+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f3584092540 con 0x7f358409b250 2024-09-16T03:09:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.226+0000 7f35817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.226+0000 7f35817fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.462+0000 7f358ac54640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:09:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.462+0000 7f358ac54640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.462+0000 7f358ac54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:04.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.462+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f355c005380 con 0x7f358409b250 2024-09-16T03:09:04.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.462+0000 7f35817fa640 1 -- 172.21.15.45:0/1304512331 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 859054833) 0x7f3574016ae0 con 0x7f358409b250 2024-09-16T03:09:04.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.462+0000 7f35817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:04.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.462+0000 7f35817fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:04.468 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:09:04.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 10 librados: watch_flush enter 2024-09-16T03:09:04.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 10 librados: watch_flush exit 2024-09-16T03:09:04.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 20 client.104169.objecter shutdown clearing up homeless session... 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 10 client.104169.objecter successfully canceled tick 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 >> v1:172.21.15.46:6812/149360280 conn(0x7f356803dfc0 legacy=0x7f3568040480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 10 monclient: shutdown 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 >> v1:172.21.15.45:6789/0 conn(0x7f358409b250 legacy=0x7f3584099360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 shutdown_connections 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 >> 172.21.15.45:0/1304512331 conn(0x7f35840a2bd0 msgr2=0x7f35840a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 shutdown_connections 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 1 -- 172.21.15.45:0/1304512331 wait complete. 2024-09-16T03:09:04.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:04.466+0000 7f358ac54640 1 librados: shutdown 2024-09-16T03:09:07.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:09:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient: get_monmap_and_config 2024-09-16T03:09:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient: build_initial_monmap 2024-09-16T03:09:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient: monmap: 2024-09-16T03:09:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:07.621365+0000 2024-09-16T03:09:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:07.621365+0000 2024-09-16T03:09:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:07.623 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:07.623 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:07.623 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:07.623 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 1 Processor -- start 2024-09-16T03:09:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 1 -- start start 2024-09-16T03:09:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient: init 2024-09-16T03:09:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient(hunting): picked mon.noname-c con 0x7fd8a0137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient(hunting): picked mon.noname-a con 0x7fd8a0138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient(hunting): picked mon.noname-b con 0x7fd8a013e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a00a4f90 con 0x7fd8a0138100 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a00a4cb0 con 0x7fd8a0137d20 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a009dc70 con 0x7fd8a013e2a0 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a7939640 10 monclient(hunting): authenticate will time out at 1238.271327s 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a5ed6640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fd8a013e2a0 legacy=0x7fd8a0140690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a4ed4640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd8a0138100 0x7fd8a013cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35576/0 (socket says 172.21.15.45:35576) 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a4ed4640 1 -- 172.21.15.45:0/530291090 learned_addr learned my addr 172.21.15.45:0/530291090 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd897fff640 1 -- 172.21.15.45:0/530291090 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3897705698 0 0) 0x7fd8a00a4f90 con 0x7fd8a0138100 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd897fff640 10 monclient(hunting): _negotiate 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd897fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd897fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd897fff640 10 monclient(hunting): my global_id is 104172 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd897fff640 1 -- 172.21.15.45:0/530291090 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd88c003530 con 0x7fd8a0138100 2024-09-16T03:09:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a56d5640 1 -- 172.21.15.45:0/530291090 >> v1:172.21.15.46:6789/0 conn(0x7fd8a0137d20 legacy=0x7fd8a00a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.618+0000 7fd8a56d5640 1 -- 172.21.15.45:0/530291090 >> v1:172.21.15.46:6789/0 conn(0x7fd8a0137d20 legacy=0x7fd8a00a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a56d5640 1 --1- 172.21.15.45:0/530291090 >> v1:172.21.15.46:6789/0 conn(0x7fd8a0137d20 0x7fd8a00a5c50 :-1 s=OPENED pgs=7875 cs=1 l=1).handle_message read tag failed 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a56d5640 1 --1- 172.21.15.45:0/530291090 >> v1:172.21.15.46:6789/0 conn(0x7fd8a0137d20 0x7fd8a00a5c50 :-1 s=OPENED pgs=7875 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.46:6789/0 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 1 -- 172.21.15.45:0/530291090 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3611605721 0 0) 0x7fd88c003530 con 0x7fd8a0138100 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 1 -- 172.21.15.45:0/530291090 >> v1:172.21.15.45:6790/0 conn(0x7fd8a013e2a0 legacy=0x7fd8a0140690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 1 -- 172.21.15.45:0/530291090 >> v1:172.21.15.46:6789/0 conn(0x7fd8a0137d20 legacy=0x7fd8a00a5c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: _finish_hunting 0 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 1 monclient: found mon.noname-a 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 1 -- 172.21.15.45:0/530291090 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a009cc50 con 0x7fd8a0138100 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: _finish_auth 0 2024-09-16T03:09:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: _check_auth_tickets 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 1 -- 172.21.15.45:0/530291090 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd8900016a0 con 0x7fd8a0138100 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 5 monclient: authenticate success, global_id 104172 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: dump: 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 1 -- 172.21.15.45:0/530291090 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8900018c0 con 0x7fd8a0138100 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 4 set_mon_vals no callback set 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 1 -- 172.21.15.45:0/530291090 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd890001d10 con 0x7fd8a0138100 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd897fff640 10 monclient: dump: 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: shutdown 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- 172.21.15.45:0/530291090 >> v1:172.21.15.45:6789/0 conn(0x7fd8a0138100 legacy=0x7fd8a013cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- 172.21.15.45:0/530291090 shutdown_connections 2024-09-16T03:09:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- 172.21.15.45:0/530291090 >> 172.21.15.45:0/530291090 conn(0x7fd8a00aa110 msgr2=0x7fd8a00aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:07.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- 172.21.15.45:0/530291090 shutdown_connections 2024-09-16T03:09:07.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- 172.21.15.45:0/530291090 wait complete. 2024-09-16T03:09:07.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: build_initial_monmap 2024-09-16T03:09:07.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: monmap: 2024-09-16T03:09:07.635 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 librados: starting msgr at 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 librados: starting objecter 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 Processor -- start 2024-09-16T03:09:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- start start 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 librados: setting wanted keys 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 librados: calling monclient init 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: init 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient(hunting): picked mon.noname-a con 0x7fd8a0137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient(hunting): picked mon.noname-b con 0x7fd8a0138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient(hunting): picked mon.noname-c con 0x7fd8a013e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a0141b60 con 0x7fd8a0137d20 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a0138a50 con 0x7fd8a0138100 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a00a2b10 con 0x7fd8a013e2a0 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient(hunting): authenticate will time out at 1238.274859s 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a5ed6640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fd8a013e2a0 legacy=0x7fd8a01d4d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a56d5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd8a0137d20 0x7fd8a00a1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35592/0 (socket says 172.21.15.45:35592) 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a56d5640 1 -- 172.21.15.45:0/1255948665 learned_addr learned my addr 172.21.15.45:0/1255948665 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 client.?.objecter ms_handle_connect 0x7fd8a0137d20 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3233546807 0 0) 0x7fd8a0141b60 con 0x7fd8a0137d20 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient(hunting): my global_id is 104175 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd884004ec0 con 0x7fd8a0137d20 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 client.?.objecter ms_handle_connect 0x7fd8a0138100 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3156397391 0 0) 0x7fd884004ec0 con 0x7fd8a0137d20 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8a0141b60 con 0x7fd8a0137d20 2024-09-16T03:09:07.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd89c001a30 con 0x7fd8a0137d20 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient(hunting): dump: 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3722577757 0 0) 0x7fd8a0141b60 con 0x7fd8a0137d20 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 >> v1:172.21.15.45:6790/0 conn(0x7fd8a013e2a0 legacy=0x7fd8a01d4d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 >> v1:172.21.15.46:6789/0 conn(0x7fd8a0138100 legacy=0x7fd8a00a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 monclient: found mon.a 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a00a3260 con 0x7fd8a0137d20 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient: _finish_auth 0 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 5 monclient: authenticate success, global_id 104175 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: _renew_subs 2024-09-16T03:09:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a00a2f90 con 0x7fd8a0137d20 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 client.104175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: _renew_subs 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8a00a2120 con 0x7fd8a0137d20 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8a7939640 1 librados: init done 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd89c001cf0 con 0x7fd8a0137d20 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd89c003a50 con 0x7fd8a0137d20 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd895ffb640 10 monclient: dump: 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.622+0000 7fd8977fe640 4 set_mon_vals no callback set 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd8a7939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd8a7939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd8a7939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd880005090 con 0x7fd8a0137d20 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fd89c003c70 con 0x7fd8a0137d20 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(184..184 src has 1..184) ==== 5975+0+0 (unknown 1295916952 0 0) 0x7fd89c04bbd0 con 0x7fd8a0137d20 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd895ffb640 10 client.104175.objecter ms_dispatch 0x7fd8a0000ba0 osd_map(184..184 src has 1..184) 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd895ffb640 3 client.104175.objecter handle_osd_map got epochs [184,184] > 0 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd895ffb640 3 client.104175.objecter handle_osd_map decoding full epoch 184 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd895ffb640 20 client.104175.objecter dump_active .. 0 homeless 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd8a4ed4640 10 monclient: get_auth_request con 0x7fd88403ded0 auth_method 0 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.626+0000 7fd895ffb640 10 client.104175.objecter ms_handle_connect 0x7fd88403ded0 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.630+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd89c016ba0 con 0x7fd8a0137d20 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.630+0000 7fd895ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:07.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.630+0000 7fd895ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:07.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd8a7939640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:09:07.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd8a7939640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:07.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd8a7939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:07.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd880005380 con 0x7fd8a0137d20 2024-09-16T03:09:07.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd895ffb640 1 -- 172.21.15.45:0/1255948665 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 204904433) 0x7fd89c01af80 con 0x7fd8a0137d20 2024-09-16T03:09:07.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd895ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:07.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd895ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:07.867 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:09:07.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd8a7939640 10 librados: watch_flush enter 2024-09-16T03:09:07.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd8a7939640 10 librados: watch_flush exit 2024-09-16T03:09:07.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd8a7939640 20 client.104175.objecter shutdown clearing up homeless session... 2024-09-16T03:09:07.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.862+0000 7fd8a7939640 10 client.104175.objecter successfully canceled tick 2024-09-16T03:09:07.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.866+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 >> v1:172.21.15.46:6812/149360280 conn(0x7fd88403ded0 legacy=0x7fd884040390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:07.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.866+0000 7fd8a7939640 10 monclient: shutdown 2024-09-16T03:09:07.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.866+0000 7fd8a7939640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:07.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.866+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 >> v1:172.21.15.45:6789/0 conn(0x7fd8a0137d20 legacy=0x7fd8a00a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:07.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.866+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 shutdown_connections 2024-09-16T03:09:07.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.866+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 >> 172.21.15.45:0/1255948665 conn(0x7fd8a00aa110 msgr2=0x7fd8a0130a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:07.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.866+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 shutdown_connections 2024-09-16T03:09:07.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.866+0000 7fd8a7939640 1 -- 172.21.15.45:0/1255948665 wait complete. 2024-09-16T03:09:07.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:07.866+0000 7fd8a7939640 1 librados: shutdown 2024-09-16T03:09:10.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:09:11.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient: get_monmap_and_config 2024-09-16T03:09:11.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient: build_initial_monmap 2024-09-16T03:09:11.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient: monmap: 2024-09-16T03:09:11.018 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:11.019 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:11.019 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:11.017680+0000 2024-09-16T03:09:11.019 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:11.017680+0000 2024-09-16T03:09:11.019 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:11.019 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:11.019 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:11.019 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:11.019 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:11.019 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:11.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 1 Processor -- start 2024-09-16T03:09:11.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 1 -- start start 2024-09-16T03:09:11.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient: init 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient(hunting): picked mon.noname-b con 0x7fe3b409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient(hunting): picked mon.noname-c con 0x7fe3b409b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient(hunting): picked mon.noname-a con 0x7fe3b413b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3b409cc20 con 0x7fe3b413b000 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3b409d820 con 0x7fe3b409b250 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3b409b6a0 con 0x7fe3b409b8e0 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b8cb8640 10 monclient(hunting): authenticate will time out at 1241.667770s 2024-09-16T03:09:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b259c640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe3b409bcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:11.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b2d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe3b413b000 0x7fe3b413d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50020/0 (socket says 172.21.15.45:50020) 2024-09-16T03:09:11.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.014+0000 7fe3b2d9d640 1 -- 172.21.15.45:0/2802989409 learned_addr learned my addr 172.21.15.45:0/2802989409 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:11.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.218+0000 7fe3b259c640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe3b409bcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:11.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:11.622+0000 7fe3b259c640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe3b409bcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:12.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:12.426+0000 7fe3b259c640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe3b409bcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:14.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.014+0000 7fe3b0d99640 10 monclient(hunting): tick 2024-09-16T03:09:14.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.014+0000 7fe3b0d99640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:09:14.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.014+0000 7fe3b0d99640 1 monclient(hunting): continuing hunt 2024-09-16T03:09:14.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.014+0000 7fe3b0d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:09:14.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.014+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe3b409bcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.014+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 legacy=0x7fe3b41398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.014+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6789/0 conn(0x7fe3b413b000 legacy=0x7fe3b413d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.014+0000 7fe3b0d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.014+0000 7fe3b0d99640 10 monclient(hunting): picked mon.noname-a con 0x7fe3940036d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3b0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fe3b409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3b0d99640 10 monclient(hunting): picked mon.noname-c con 0x7fe3b409b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3b40a21f0 con 0x7fe3940036d0 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3b409b6a0 con 0x7fe3b409b250 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe394010400 con 0x7fe3b409b8e0 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3b0d99640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3abfff640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe394009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3b2d9d640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 legacy=0x7fe39400cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3b2d9d640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 legacy=0x7fe39400cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:09:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.018+0000 7fe3b2d9d640 1 --1- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 0x7fe39400cc00 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-16T03:09:14.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.222+0000 7fe3abfff640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe394009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:14.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:14.626+0000 7fe3abfff640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe394009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:15.430+0000 7fe3abfff640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe394009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:17.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 10 monclient(hunting): tick 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 1 monclient(hunting): continuing hunt 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe394009420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 legacy=0x7fe39400cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6789/0 conn(0x7fe3940036d0 legacy=0x7fe394005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fe3b409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 10 monclient(hunting): picked mon.noname-a con 0x7fe394010e10 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 10 monclient(hunting): picked mon.noname-c con 0x7fe3b409b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe394011700 con 0x7fe394010e10 2024-09-16T03:09:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe394010400 con 0x7fe3b409b250 2024-09-16T03:09:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 1 -- 172.21.15.45:0/2802989409 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe394004b20 con 0x7fe3b409b8e0 2024-09-16T03:09:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b0d99640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b259c640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe394010700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b2d9d640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 legacy=0x7fe39400cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T03:09:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b2d9d640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 legacy=0x7fe39400cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:09:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b2d9d640 1 --1- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 0x7fe39400cc00 :-1 s=OPENED pgs=7881 cs=1 l=1).handle_message read tag failed 2024-09-16T03:09:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b2d9d640 1 --1- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 0x7fe39400cc00 :-1 s=OPENED pgs=7881 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:09:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.018+0000 7fe3b1d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.46:6789/0 2024-09-16T03:09:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.222+0000 7fe3b259c640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe394010700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:17.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:17.626+0000 7fe3b259c640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe394010700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:18.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.218+0000 7fe3b1d9b640 1 -- 172.21.15.45:0/2802989409 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688853127 0 0) 0x7fe394011700 con 0x7fe394010e10 2024-09-16T03:09:18.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.218+0000 7fe3b1d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.218+0000 7fe3b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.218+0000 7fe3b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.218+0000 7fe3b1d9b640 10 monclient(hunting): my global_id is 104178 2024-09-16T03:09:18.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.218+0000 7fe3b1d9b640 1 -- 172.21.15.45:0/2802989409 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3a0003530 con 0x7fe394010e10 2024-09-16T03:09:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 1 -- 172.21.15.45:0/2802989409 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1480096685 0 0) 0x7fe3a0003530 con 0x7fe394010e10 2024-09-16T03:09:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe394010700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:09:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.46:6789/0 conn(0x7fe3b409b250 legacy=0x7fe39400cc00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:09:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 1 monclient: found mon.noname-a 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 1 -- 172.21.15.45:0/2802989409 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe394005d00 con 0x7fe394010e10 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: _finish_auth 0 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 1 -- 172.21.15.45:0/2802989409 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe39c0016d0 con 0x7fe394010e10 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b8cb8640 5 monclient: authenticate success, global_id 104178 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b8cb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: dump: 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b8cb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 1 -- 172.21.15.45:0/2802989409 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe39c001870 con 0x7fe394010e10 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b8cb8640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b8cb8640 4 set_mon_vals no callback set 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 1 -- 172.21.15.45:0/2802989409 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe39c001cc0 con 0x7fe394010e10 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.230+0000 7fe3b1d9b640 10 monclient: dump: 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.239 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient: shutdown 2024-09-16T03:09:18.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:18.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2802989409 >> v1:172.21.15.45:6789/0 conn(0x7fe394010e10 legacy=0x7fe394009420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2802989409 shutdown_connections 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2802989409 >> 172.21.15.45:0/2802989409 conn(0x7fe3b40a2bd0 msgr2=0x7fe3b40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2802989409 shutdown_connections 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2802989409 wait complete. 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient: build_initial_monmap 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient: monmap: 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:18.245 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 librados: starting msgr at 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 librados: starting objecter 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 Processor -- start 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 -- start start 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 librados: setting wanted keys 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 librados: calling monclient init 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient: init 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient(hunting): picked mon.noname-b con 0x7fe394010e10 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient(hunting): picked mon.noname-a con 0x7fe3b409b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient(hunting): picked mon.noname-c con 0x7fe3b409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3b409d2c0 con 0x7fe3b409b250 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3b4071be0 con 0x7fe394010e10 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3b4072480 con 0x7fe3b409b8e0 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b8cb8640 10 monclient(hunting): authenticate will time out at 1248.887864s 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.234+0000 7fe3b2d9d640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe3b41d07a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3abfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe3b409b250 0x7fe3b40739b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50048/0 (socket says 172.21.15.45:50048) 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3abfff640 1 -- 172.21.15.45:0/2068196056 learned_addr learned my addr 172.21.15.45:0/2068196056 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 client.?.objecter ms_handle_connect 0x7fe3b409b250 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 client.?.objecter ms_handle_connect 0x7fe394010e10 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71107747 0 0) 0x7fe3b409d2c0 con 0x7fe3b409b250 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): my global_id is 104181 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3940116a0 con 0x7fe3b409b250 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184420573 0 0) 0x7fe3b4071be0 con 0x7fe394010e10 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): my global_id is 22684 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3b409d2c0 con 0x7fe394010e10 2024-09-16T03:09:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1049754951 0 0) 0x7fe3940116a0 con 0x7fe3b409b250 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3b4071be0 con 0x7fe3b409b250 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe39c001c00 con 0x7fe3b409b250 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): dump: 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.249 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1769894824 0 0) 0x7fe3b4071be0 con 0x7fe3b409b250 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 >> v1:172.21.15.45:6790/0 conn(0x7fe3b409b8e0 legacy=0x7fe3b41d07a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 >> v1:172.21.15.46:6789/0 conn(0x7fe394010e10 legacy=0x7fe3b41c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient: _finish_hunting 0 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 monclient: found mon.a 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b4070c80 con 0x7fe3b409b250 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient: _finish_auth 0 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient: _check_auth_tickets 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 5 monclient: authenticate success, global_id 104181 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 10 monclient: _renew_subs 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b40740c0 con 0x7fe3b409b250 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 10 client.104181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 10 monclient: _renew_subs 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe3b41d2090 con 0x7fe3b409b250 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 1 librados: init done 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe39c0038b0 con 0x7fe3b409b250 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe39c003a60 con 0x7fe3b409b250 2024-09-16T03:09:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 monclient: dump: 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b0d99640 4 set_mon_vals no callback set 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe398005090 con 0x7fe3b409b250 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fe39c003c00 con 0x7fe3b409b250 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(185..185 src has 1..185) ==== 6188+0+0 (unknown 545366752 0 0) 0x7fe39c04b950 con 0x7fe3b409b250 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 10 client.104181.objecter ms_dispatch 0x7fe3b4000ba0 osd_map(185..185 src has 1..185) 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 3 client.104181.objecter handle_osd_map got epochs [185,185] > 0 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 3 client.104181.objecter handle_osd_map decoding full epoch 185 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3aaffd640 20 client.104181.objecter dump_active .. 0 homeless 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.238+0000 7fe3b259c640 10 monclient: get_auth_request con 0x7fe394040dd0 auth_method 0 2024-09-16T03:09:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.242+0000 7fe3aaffd640 10 client.104181.objecter ms_handle_connect 0x7fe394040dd0 2024-09-16T03:09:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.246+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fe39c016b10 con 0x7fe3b409b250 2024-09-16T03:09:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.246+0000 7fe3aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.246+0000 7fe3aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.494+0000 7fe3b8cb8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:09:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.494+0000 7fe3b8cb8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.494+0000 7fe3b8cb8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.494+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe398005380 con 0x7fe3b409b250 2024-09-16T03:09:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3aaffd640 1 -- 172.21.15.45:0/2068196056 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1585762282) 0x7fe39c016410 con 0x7fe3b409b250 2024-09-16T03:09:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:09:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:18.501 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:09:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 10 librados: watch_flush enter 2024-09-16T03:09:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 10 librados: watch_flush exit 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 20 client.104181.objecter shutdown clearing up homeless session... 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 10 client.104181.objecter successfully canceled tick 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 >> v1:172.21.15.46:6812/149360280 conn(0x7fe394040dd0 legacy=0x7fe394043290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 10 monclient: shutdown 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 >> v1:172.21.15.45:6789/0 conn(0x7fe3b409b250 legacy=0x7fe3b40739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 shutdown_connections 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 >> 172.21.15.45:0/2068196056 conn(0x7fe3b40a2bd0 msgr2=0x7fe3b409ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 shutdown_connections 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 1 -- 172.21.15.45:0/2068196056 wait complete. 2024-09-16T03:09:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.498+0000 7fe3b8cb8640 1 librados: shutdown 2024-09-16T03:09:18.521 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:09:18.521 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:09:18.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient: get_monmap_and_config 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient: build_initial_monmap 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient: monmap: 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:18.645227+0000 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:18.645227+0000 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:18.647 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:18.647 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 1 Processor -- start 2024-09-16T03:09:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 1 -- start start 2024-09-16T03:09:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient: init 2024-09-16T03:09:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:09:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient(hunting): picked mon.noname-c con 0x7f7944137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient(hunting): picked mon.noname-a con 0x7f7944138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient(hunting): picked mon.noname-b con 0x7f794413e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79440a4f90 con 0x7f7944138100 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79440a4cb0 con 0x7f7944137d20 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794409dc70 con 0x7f794413e2a0 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient(hunting): authenticate will time out at 1249.294592s 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f7949cc7640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f794413e2a0 legacy=0x7f7944140690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f7948cc5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7944138100 0x7f794413cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50062/0 (socket says 172.21.15.45:50062) 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f7948cc5640 1 -- 172.21.15.45:0/792955343 learned_addr learned my addr 172.21.15.45:0/792955343 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 -- 172.21.15.45:0/792955343 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2959410914 0 0) 0x7f79440a4f90 con 0x7f7944138100 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient(hunting): my global_id is 104184 2024-09-16T03:09:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 -- 172.21.15.45:0/792955343 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7930003530 con 0x7f7944138100 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 -- 172.21.15.45:0/792955343 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2407526664 0 0) 0x7f7930003530 con 0x7f7944138100 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 -- 172.21.15.45:0/792955343 >> v1:172.21.15.45:6790/0 conn(0x7f794413e2a0 legacy=0x7f7944140690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 -- 172.21.15.45:0/792955343 >> v1:172.21.15.46:6789/0 conn(0x7f7944137d20 legacy=0x7f79440a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 monclient: found mon.noname-a 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 -- 172.21.15.45:0/792955343 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794409cc50 con 0x7f7944138100 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: _finish_auth 0 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 5 monclient: authenticate success, global_id 104184 2024-09-16T03:09:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 -- 172.21.15.45:0/792955343 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f79340016a0 con 0x7f7944138100 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: dump: 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 -- 172.21.15.45:0/792955343 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79340018c0 con 0x7f7944138100 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 1 -- 172.21.15.45:0/792955343 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7934001d50 con 0x7f7944138100 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f793bfff640 10 monclient: dump: 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.660 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.642+0000 7f794b72a640 4 set_mon_vals no callback set 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: shutdown 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- 172.21.15.45:0/792955343 >> v1:172.21.15.45:6789/0 conn(0x7f7944138100 legacy=0x7f794413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- 172.21.15.45:0/792955343 shutdown_connections 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- 172.21.15.45:0/792955343 >> 172.21.15.45:0/792955343 conn(0x7f79440aa110 msgr2=0x7f79440aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- 172.21.15.45:0/792955343 shutdown_connections 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- 172.21.15.45:0/792955343 wait complete. 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: build_initial_monmap 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: monmap: 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:18.662 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 librados: starting msgr at 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 librados: starting objecter 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 Processor -- start 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- start start 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 librados: setting wanted keys 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 librados: calling monclient init 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: init 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient(hunting): picked mon.noname-b con 0x7f7944137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient(hunting): picked mon.noname-c con 0x7f7944138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient(hunting): picked mon.noname-a con 0x7f794413e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:18.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794413a7b0 con 0x7f794413e2a0 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79440a1300 con 0x7f7944137d20 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794413a4d0 con 0x7f7944138100 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient(hunting): authenticate will time out at 1249.297771s 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7948cc5640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f7944138100 legacy=0x7f79440a31b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7949cc7640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f794413e2a0 0x7f79440a38c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50068/0 (socket says 172.21.15.45:50068) 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7949cc7640 1 -- 172.21.15.45:0/4080554275 learned_addr learned my addr 172.21.15.45:0/4080554275 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 client.?.objecter ms_handle_connect 0x7f794413e2a0 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1429116242 0 0) 0x7f794413a7b0 con 0x7f794413e2a0 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): my global_id is 104187 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7928004ec0 con 0x7f794413e2a0 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 client.?.objecter ms_handle_connect 0x7f7944137d20 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2538138438 0 0) 0x7f79440a1300 con 0x7f7944137d20 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): my global_id is 22690 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f794413a7b0 con 0x7f7944137d20 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2416815288 0 0) 0x7f7928004ec0 con 0x7f794413e2a0 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79440a1300 con 0x7f794413e2a0 2024-09-16T03:09:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f793c004440 con 0x7f794413e2a0 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): dump: 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.666 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3653009134 0 0) 0x7f79440a1300 con 0x7f794413e2a0 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 >> v1:172.21.15.45:6790/0 conn(0x7f7944138100 legacy=0x7f79440a31b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 >> v1:172.21.15.46:6789/0 conn(0x7f7944137d20 legacy=0x7f79440a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 monclient: found mon.a 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7944139560 con 0x7f794413e2a0 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient: _finish_auth 0 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:18.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 5 monclient: authenticate success, global_id 104187 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: _renew_subs 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f79441d5d80 con 0x7f794413e2a0 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 client.104187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: _renew_subs 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f79441d6120 con 0x7f794413e2a0 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 librados: init done 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f793c004660 con 0x7f794413e2a0 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f793c004a30 con 0x7f794413e2a0 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f793b7fe640 4 set_mon_vals no callback set 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 monclient: dump: 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79240065d0 con 0x7f7924002870 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 client.104187.objecter ms_handle_connect 0x7f7924002870 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 10 client.104187.objecter resend_mon_ops 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.646+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f793c004c50 con 0x7f794413e2a0 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(185..185 src has 1..185) ==== 6188+0+0 (unknown 545366752 0 0) 0x7f793c04be20 con 0x7f794413e2a0 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 client.104187.objecter ms_dispatch 0x7f7944000ba0 osd_map(185..185 src has 1..185) 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 3 client.104187.objecter handle_osd_map got epochs [185,185] > 0 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 3 client.104187.objecter handle_osd_map decoding full epoch 185 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 20 client.104187.objecter dump_active .. 0 homeless 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192271061 0 0) 0x7f79240065d0 con 0x7f7924002870 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 monclient(hunting): my global_id is 104190 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79440a1300 con 0x7f7924002870 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7948cc5640 10 monclient: get_auth_request con 0x7f792803dfd0 auth_method 0 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2763210897 0 0) 0x7f79440a1300 con 0x7f7924002870 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79240065d0 con 0x7f7924002870 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f79400017c0 con 0x7f7924002870 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 monclient: dump: 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4169848444 0 0) 0x7f79240065d0 con 0x7f7924002870 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f792400a170 con 0x7f7924002870 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 client.104187.objecter ms_handle_connect 0x7f792803dfd0 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7940001890 con 0x7f7924002870 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 >> v1:172.21.15.45:6789/0 conn(0x7f7924002870 legacy=0x7f7924004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.650+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 >> v1:172.21.15.45:6789/0 conn(0x7f7924002870 legacy=0x7f7924004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:18.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f794b72a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:09:18.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7924009e60 con 0x7f7924009120 2024-09-16T03:09:18.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 10 client.104187.objecter ms_handle_connect 0x7f7924009120 2024-09-16T03:09:18.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 10 client.104187.objecter resend_mon_ops 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392447030 0 0) 0x7f7924009e60 con 0x7f7924009120 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 10 monclient(hunting): my global_id is 104193 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79240065d0 con 0x7f7924009120 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1334317258 0 0) 0x7f79240065d0 con 0x7f7924009120 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7924009e60 con 0x7f7924009120 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f79400037c0 con 0x7f7924009120 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 10 monclient: dump: 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2296937551 0 0) 0x7f7924009e60 con 0x7f7924009120 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.654+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f792400c400 con 0x7f7924009120 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4125 (unknown 0 0 118096011) 0x7f7940005a80 con 0x7f7924009120 2024-09-16T03:09:18.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f7939ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f7939ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 >> v1:172.21.15.45:6789/0 conn(0x7f7924009120 legacy=0x7f79240095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f7939ffb640 1 -- 172.21.15.45:0/4080554275 >> v1:172.21.15.45:6789/0 conn(0x7f7924009120 legacy=0x7f79240095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 24, 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 17759, 2024-09-16T03:09:18.677 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: 1 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:09:18.680 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:18.681 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:09:18.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:09:18.683 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:18.684 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:18.685 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:18.686 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:18.687 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:09:18.688 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 10 librados: watch_flush enter 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 10 librados: watch_flush exit 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 20 client.104187.objecter shutdown clearing up homeless session... 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 10 client.104187.objecter successfully canceled tick 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 >> v1:172.21.15.46:6812/149360280 conn(0x7f792803dfd0 legacy=0x7f7928040490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 10 monclient: shutdown 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 >> v1:172.21.15.45:6789/0 conn(0x7f794413e2a0 legacy=0x7f79440a38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f7949cc7640 1 -- 172.21.15.45:0/4080554275 reap_dead start 2024-09-16T03:09:18.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f7949cc7640 1 -- 172.21.15.45:0/4080554275 reap_dead start 2024-09-16T03:09:18.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 shutdown_connections 2024-09-16T03:09:18.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 >> 172.21.15.45:0/4080554275 conn(0x7f79440aa110 msgr2=0x7f79440a06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:18.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 shutdown_connections 2024-09-16T03:09:18.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 1 -- 172.21.15.45:0/4080554275 wait complete. 2024-09-16T03:09:18.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.658+0000 7f794b72a640 1 librados: shutdown 2024-09-16T03:09:18.691 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:09:18.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:09:18.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:09:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient: get_monmap_and_config 2024-09-16T03:09:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient: build_initial_monmap 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient: monmap: 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:09:18.808778+0000 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:09:18.808778+0000 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:09:18.810 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 1 Processor -- start 2024-09-16T03:09:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 1 -- start start 2024-09-16T03:09:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient: init 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient(hunting): picked mon.noname-a con 0x7fbe00135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient(hunting): picked mon.noname-b con 0x7fbe0009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient(hunting): picked mon.noname-c con 0x7fbe0013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0009f7e0 con 0x7fbe00135950 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0009b160 con 0x7fbe0013c010 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0009b440 con 0x7fbe0009fc40 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient(hunting): authenticate will time out at 1249.458624s 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdfffff640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fbe0009fc40 legacy=0x7fbe0013a8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe049bc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbe00135950 0x7fbe00135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50092/0 (socket says 172.21.15.45:50092) 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe049bc640 1 -- 172.21.15.45:0/65605515 learned_addr learned my addr 172.21.15.45:0/65605515 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2991112681 0 0) 0x7fbe0009f7e0 con 0x7fbe00135950 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient(hunting): my global_id is 104196 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde8003530 con 0x7fbe00135950 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 430986926 0 0) 0x7fbe0009b160 con 0x7fbe0013c010 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient(hunting): my global_id is 22693 2024-09-16T03:09:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe0009f7e0 con 0x7fbe0013c010 2024-09-16T03:09:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2288233917 0 0) 0x7fbde8003530 con 0x7fbe00135950 2024-09-16T03:09:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 >> v1:172.21.15.45:6790/0 conn(0x7fbe0009fc40 legacy=0x7fbe0013a8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:09:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 >> v1:172.21.15.46:6789/0 conn(0x7fbe0013c010 legacy=0x7fbe0013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:09:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 monclient: found mon.noname-a 2024-09-16T03:09:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe00136440 con 0x7fbe00135950 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: _finish_auth 0 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdf00016a0 con 0x7fbe00135950 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 5 monclient: authenticate success, global_id 104196 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: dump: 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdf00018c0 con 0x7fbe00135950 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:18.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 1 -- 172.21.15.45:0/65605515 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdf0001d10 con 0x7fbe00135950 2024-09-16T03:09:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbdff7fe640 10 monclient: dump: 2024-09-16T03:09:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 10 monclient: get_monmap_and_config success 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.806+0000 7fbe06c20640 4 set_mon_vals no callback set 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: shutdown 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- 172.21.15.45:0/65605515 >> v1:172.21.15.45:6789/0 conn(0x7fbe00135950 legacy=0x7fbe00135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- 172.21.15.45:0/65605515 shutdown_connections 2024-09-16T03:09:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- 172.21.15.45:0/65605515 >> 172.21.15.45:0/65605515 conn(0x7fbe000a2bd0 msgr2=0x7fbe000a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- 172.21.15.45:0/65605515 shutdown_connections 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- 172.21.15.45:0/65605515 wait complete. 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: build_initial_monmap 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: monmap: 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:09:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 librados: starting msgr at 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 librados: starting objecter 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 Processor -- start 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- start start 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 librados: setting wanted keys 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 librados: calling monclient init 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: init 2024-09-16T03:09:18.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: _reopen_session rank -1 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient(hunting): picked mon.noname-a con 0x7fbe0009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient(hunting): picked mon.noname-b con 0x7fbe00135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient(hunting): picked mon.noname-c con 0x7fbe0013c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0009d2c0 con 0x7fbe0009fc40 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe000735e0 con 0x7fbe00135950 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0009d8c0 con 0x7fbe0013c010 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient(hunting): _renew_subs 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient(hunting): authenticate will time out at 1249.461915s 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe051bd640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fbe0013c010 legacy=0x7fbe001d4ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe049bc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbe0009fc40 0x7fbe000727c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50102/0 (socket says 172.21.15.45:50102) 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe049bc640 1 -- 172.21.15.45:0/1167674406 learned_addr learned my addr 172.21.15.45:0/1167674406 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 client.?.objecter ms_handle_connect 0x7fbe0009fc40 2024-09-16T03:09:18.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 840533328 0 0) 0x7fbe00136440 con 0x7fbe0009fc40 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): my global_id is 104199 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdec004ec0 con 0x7fbe0009fc40 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 client.?.objecter ms_handle_connect 0x7fbe00135950 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 439413019 0 0) 0x7fbdec004ec0 con 0x7fbe0009fc40 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbe00136440 con 0x7fbe0009fc40 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdf0001c10 con 0x7fbe0009fc40 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:09:18.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): dump: 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1059759649 0 0) 0x7fbe000735e0 con 0x7fbe00135950 2024-09-16T03:09:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient(hunting): my global_id is 22696 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdec004ec0 con 0x7fbe00135950 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2850615300 0 0) 0x7fbe00136440 con 0x7fbe0009fc40 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 >> v1:172.21.15.45:6790/0 conn(0x7fbe0013c010 legacy=0x7fbe001d4ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 >> v1:172.21.15.46:6789/0 conn(0x7fbe00135950 legacy=0x7fbe00072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient: _finish_hunting 0 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 monclient: found mon.a 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0009eb10 con 0x7fbe0009fc40 2024-09-16T03:09:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient: _finish_auth 0 2024-09-16T03:09:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:09:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:09:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 5 monclient: authenticate success, global_id 104199 2024-09-16T03:09:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: _renew_subs 2024-09-16T03:09:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0009edf0 con 0x7fbe0009fc40 2024-09-16T03:09:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 client.104199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:09:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: _renew_subs 2024-09-16T03:09:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbe0013b0b0 con 0x7fbe0009fc40 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 librados: init done 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdf0004c80 con 0x7fbe0009fc40 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfeffd640 4 set_mon_vals no callback set 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdf0004f30 con 0x7fbe0009fc40 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 10 monclient: dump: 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbde00065d0 con 0x7fbde0002870 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.810+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fbdf0005150 con 0x7fbe0009fc40 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(185..185 src has 1..185) ==== 6188+0+0 (unknown 545366752 0 0) 0x7fbdf004bbc0 con 0x7fbe0009fc40 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 client.104199.objecter ms_dispatch 0x7fbe00000ba0 osd_map(185..185 src has 1..185) 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 3 client.104199.objecter handle_osd_map got epochs [185,185] > 0 2024-09-16T03:09:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 3 client.104199.objecter handle_osd_map decoding full epoch 185 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 20 client.104199.objecter dump_active .. 0 homeless 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 client.104199.objecter ms_handle_connect 0x7fbde0002870 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 client.104199.objecter resend_mon_ops 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbe051bd640 10 monclient: get_auth_request con 0x7fbdec03e030 auth_method 0 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1808526647 0 0) 0x7fbde00065d0 con 0x7fbde0002870 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 monclient(hunting): my global_id is 22699 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe00136440 con 0x7fbde0002870 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 client.104199.objecter ms_handle_connect 0x7fbdec03e030 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4209713196 0 0) 0x7fbe00136440 con 0x7fbde0002870 2024-09-16T03:09:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbde00065d0 con 0x7fbde0002870 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdf4004130 con 0x7fbde0002870 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 monclient: dump: 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.837 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3246695621 0 0) 0x7fbde00065d0 con 0x7fbde0002870 2024-09-16T03:09:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbde000a170 con 0x7fbde0002870 2024-09-16T03:09:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbdf40063f0 con 0x7fbde0002870 2024-09-16T03:09:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:09:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:09:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 >> v1:172.21.15.46:6789/0 conn(0x7fbde0002870 legacy=0x7fbde0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.814+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 >> v1:172.21.15.46:6789/0 conn(0x7fbde0002870 legacy=0x7fbde0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:18.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:09:18.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbde0009e60 con 0x7fbde0009120 2024-09-16T03:09:18.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 client.104199.objecter ms_handle_connect 0x7fbde0009120 2024-09-16T03:09:18.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 client.104199.objecter resend_mon_ops 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2883312922 0 0) 0x7fbde0009e60 con 0x7fbde0009120 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 monclient(hunting): my global_id is 22702 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde00065d0 con 0x7fbde0009120 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2205633210 0 0) 0x7fbde00065d0 con 0x7fbde0009120 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbde0009e60 con 0x7fbde0009120 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdf4005c00 con 0x7fbde0009120 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 monclient: dump: 2024-09-16T03:09:18.840 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 848908835 0 0) 0x7fbde0009e60 con 0x7fbde0009120 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbde000c400 con 0x7fbde0009120 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4291 (unknown 0 0 320383371) 0x7fbdf4006680 con 0x7fbde0009120 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:09:18.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 >> v1:172.21.15.46:6789/0 conn(0x7fbde0009120 legacy=0x7fbde00095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbdfd7fa640 1 -- 172.21.15.45:0/1167674406 >> v1:172.21.15.46:6789/0 conn(0x7fbde0009120 legacy=0x7fbde00095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 24, 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 280628, 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:09:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: 1 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:09:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:09:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:09:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:09:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:09:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:09:18.852 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.853 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:09:18.854 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:09:18.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:09:18.855 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:09:18.855 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:09:18.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 10 librados: watch_flush enter 2024-09-16T03:09:18.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 10 librados: watch_flush exit 2024-09-16T03:09:18.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 20 client.104199.objecter shutdown clearing up homeless session... 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 10 client.104199.objecter successfully canceled tick 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 >> v1:172.21.15.46:6812/149360280 conn(0x7fbdec03e030 legacy=0x7fbdec0404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 10 monclient: shutdown 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 >> v1:172.21.15.45:6789/0 conn(0x7fbe0009fc40 legacy=0x7fbe000727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe051bd640 1 -- 172.21.15.45:0/1167674406 reap_dead start 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe051bd640 1 -- 172.21.15.45:0/1167674406 reap_dead start 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 shutdown_connections 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 >> 172.21.15.45:0/1167674406 conn(0x7fbe000a2bd0 msgr2=0x7fbe0013ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 shutdown_connections 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 1 -- 172.21.15.45:0/1167674406 wait complete. 2024-09-16T03:09:18.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:09:18.822+0000 7fbe06c20640 1 librados: shutdown 2024-09-16T03:09:18.858 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:09:18.858 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:09:50.951 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosA api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: seed 28678 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (356 ms) 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10263 ms) 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (43 ms) 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10662 ms total) 2024-09-16T03:09:50.952 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (6960 ms) 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8048 ms) 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10002 ms) 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: my_snaps [3] 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: my_snaps [4,3] 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: my_snaps [5,4,3] 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-09-16T03:09:50.953 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: promoting some heads 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: promoting from clones for snap 3 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: done waiting 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (54010 ms) 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10016 ms) 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7982 ms) 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-16T03:09:50.954 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8024 ms) 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (7975 ms) 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10002 ms) 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9019 ms) 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9997 ms) 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13012 ms) 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7995 ms) 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-16T03:09:50.955 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (21894 ms) 2024-09-16T03:09:50.956 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-16T03:09:50.956 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12761 ms) 2024-09-16T03:09:50.956 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-16T03:09:50.956 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7954 ms) 2024-09-16T03:09:50.956 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-16T03:09:50.956 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8012 ms) 2024-09-16T03:09:50.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitS 2024-09-16T03:09:50.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6978 ms) 2024-09-16T03:09:50.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-16T03:09:50.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7004 ms) 2024-09-16T03:09:50.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-16T03:09:50.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7030 ms) 2024-09-16T03:09:50.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-16T03:09:50.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6978 ms) 2024-09-16T03:09:50.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6988 ms) 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7015 ms) 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7022 ms) 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7000 ms) 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6968 ms) 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7026 ms) 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-16T03:09:50.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6801 ms) 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (21102 ms) 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6703 ms) 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6995 ms) 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6999 ms) 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7025 ms) 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (125634 ms total) 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [----------] Global test environment tear-down 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (229499 ms total) 2024-09-16T03:09:50.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-16T03:10:06.632 INFO:tasks.workunit.client.0.smithi046.stdout:etRead 2024-09-16T03:10:06.632 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T03:10:06.632 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: ok, hit_set contains 267:602f83fe:::foo:head 2024-09-16T03:10:06.633 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9011 ms) 2024-09-16T03:10:06.633 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-16T03:10:06.633 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg_num = 32 2024-09-16T03:10:06.633 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 0 ls 1726456206,0 2024-09-16T03:10:06.633 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 1 ls 1726456206,0 2024-09-16T03:10:06.634 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 2 ls 1726456206,0 2024-09-16T03:10:06.634 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 3 ls 1726456206,0 2024-09-16T03:10:06.634 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 4 ls 1726456206,0 2024-09-16T03:10:06.634 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 5 ls 1726456206,0 2024-09-16T03:10:06.634 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 6 ls 1726456206,0 2024-09-16T03:10:06.635 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 7 ls 1726456206,0 2024-09-16T03:10:06.635 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 8 ls 1726456206,0 2024-09-16T03:10:06.635 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 9 ls 1726456206,0 2024-09-16T03:10:06.635 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 10 ls 1726456206,0 2024-09-16T03:10:06.635 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 11 ls 1726456206,0 2024-09-16T03:10:06.635 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 12 ls 1726456206,0 2024-09-16T03:10:06.636 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 13 ls 1726456206,0 2024-09-16T03:10:06.636 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 14 ls 1726456206,0 2024-09-16T03:10:06.636 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 15 ls 1726456206,0 2024-09-16T03:10:06.636 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 16 ls 1726456206,0 2024-09-16T03:10:06.636 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 17 ls 1726456206,0 2024-09-16T03:10:06.636 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 18 ls 1726456206,0 2024-09-16T03:10:06.636 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 19 ls 1726456206,0 2024-09-16T03:10:06.636 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 20 ls 1726456206,0 2024-09-16T03:10:06.636 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 21 ls 1726456206,0 2024-09-16T03:10:06.637 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 22 ls 1726456206,0 2024-09-16T03:10:06.637 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 23 ls 1726456206,0 2024-09-16T03:10:06.637 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 24 ls 1726456206,0 2024-09-16T03:10:06.637 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 25 ls 1726456206,0 2024-09-16T03:10:06.637 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 26 ls 1726456206,0 2024-09-16T03:10:06.640 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 27 ls 1726456206,0 2024-09-16T03:10:06.640 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 28 ls 1726456206,0 2024-09-16T03:10:06.640 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 29 ls 1726456206,0 2024-09-16T03:10:06.640 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 30 ls 1726456206,0 2024-09-16T03:10:06.640 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg 31 ls 1726456206,0 2024-09-16T03:10:06.640 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: pg_num = 32 2024-09-16T03:10:06.640 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:6cac518f:::0:head 2024-09-16T03:10:06.641 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:02547ec2:::1:head 2024-09-16T03:10:06.641 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:f905c69b:::2:head 2024-09-16T03:10:06.641 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:cfc208b3:::3:head 2024-09-16T03:10:06.641 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d83876eb:::4:head 2024-09-16T03:10:06.641 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:b29083e3:::5:head 2024-09-16T03:10:06.641 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c4fdafeb:::6:head 2024-09-16T03:10:06.641 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:5c6b0b28:::7:head 2024-09-16T03:10:06.641 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:bd63b0f1:::8:head 2024-09-16T03:10:06.641 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:e960b815:::9:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:52ea6a34:::10:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:89d3ae78:::11:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:de5d7c5f:::12:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:566253c9:::13:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:62a1935d:::14:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:863748b0:::15:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:3958e169:::16:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:4d4dabf9:::17:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:8391935d:::18:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:28883081:::19:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:69259c59:::20:head 2024-09-16T03:10:06.642 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:4bdb80b7:::21:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:a11c5d71:::22:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:271af37b:::23:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:95b121be:::24:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:58d1031b:::25:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:0a050783:::26:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c709704c:::27:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:cbe56eaf:::28:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:86b4b162:::29:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:70d89383:::30:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:dd450c7c:::31:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:6d5729b1:::32:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c388f3fb:::33:head 2024-09-16T03:10:06.643 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:56cfea31:::34:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:9dbc1bf7:::35:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:40b74ccd:::36:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:4d5aaf42:::37:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:920f362c:::38:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:6cc53222:::39:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:9cad833f:::40:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:1ea84d41:::41:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c4480ef6:::42:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:a694361e:::43:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d1bd33e9:::44:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:ddc2cd5d:::45:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:2b782207:::46:head 2024-09-16T03:10:06.644 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:7b187fca:::47:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:90ecdf6f:::48:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:a5ed95fe:::49:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:ea0eaa55:::50:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:f33ef17b:::51:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:a0d1b2f6:::52:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:60c5229e:::53:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:edcbc575:::54:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:102cf253:::55:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:efb7fb0b:::56:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:50d0a326:::57:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d4dc5daf:::58:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:3a130462:::59:head 2024-09-16T03:10:06.645 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:ec87ed71:::60:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d5bc9454:::61:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:3ddfe313:::62:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:7c2816b9:::63:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:47e00e4d:::64:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c6410c18:::65:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:b48ed237:::66:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:cd63ad31:::67:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:b179e92b:::68:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:0d9f741a:::69:head 2024-09-16T03:10:06.646 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:6d3352ae:::70:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c6d5c19e:::71:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:bc4729c3:::72:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:77e930b9:::73:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:0abeecfd:::74:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:b7c37e15:::75:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:b6378398:::76:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:02bd68de:::77:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:cc795d2d:::78:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:630d4fea:::79:head 2024-09-16T03:10:06.647 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:e0d29ef5:::80:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:fd6f13d2:::81:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:606461d5:::82:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:eadbdc43:::83:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:8761d0bb:::84:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:9ef0186f:::85:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:e0d41294:::86:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:961de695:::87:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:1423148f:::88:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:633a8fa2:::89:head 2024-09-16T03:10:06.648 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:a8653809:::90:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:3dac8b33:::91:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:35aad435:::92:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:f6dcc343:::93:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:dbbdad87:::94:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:1cb48ce0:::95:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:03cd461c:::96:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:17a4ea99:::97:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:9993c9a7:::98:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:6394211c:::99:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:94c7ae57:::100:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:6fdee5bb:::101:head 2024-09-16T03:10:06.649 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:9a477fd1:::102:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:eb850916:::103:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:affc56b9:::104:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:b42dc814:::105:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:f319f8f0:::106:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:9a40b9de:::107:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:8b524f28:::108:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:e3de589f:::109:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:90f90a5b:::110:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:a7b4f1d7:::111:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:af51766e:::112:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:b6f90bd1:::113:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:e0261208:::114:head 2024-09-16T03:10:06.650 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c9569ef7:::115:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:61bebe50:::116:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:fe93412b:::117:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d3d38bee:::118:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:3100ba0c:::119:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d0560ada:::120:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:f0ea8b35:::121:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:766f231a:::122:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:a07a2582:::123:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:bd7c6b3a:::124:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:fb2ddaff:::125:head 2024-09-16T03:10:06.651 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:4408e1fe:::126:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:ee1df7a7:::127:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c3002909:::128:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:4f48ffa9:::129:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:edf38733:::130:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c08425c0:::131:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:5f902d98:::132:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:41ea2c93:::133:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:813cee13:::134:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:0131818d:::135:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:26ba5a85:::136:head 2024-09-16T03:10:06.652 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:381b8a5a:::137:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:28797e47:::138:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:bfca7f22:::139:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:36807075:::140:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:80b03975:::141:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:5c15709b:::142:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:f39ea15e:::143:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:ea992956:::144:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:48887b1c:::145:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:9f24a9dd:::146:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:987f100b:::147:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d2dd3581:::148:head 2024-09-16T03:10:06.653 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:7fed1808:::149:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c80b70e9:::150:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:85ed90f9:::151:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:36428b24:::152:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d044c34a:::153:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:7c18bf58:::154:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d1c21232:::155:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:a7a3c575:::156:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:87da0633:::157:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d5ac3822:::158:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:3f20522d:::159:head 2024-09-16T03:10:06.654 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:6ca26563:::160:head 2024-09-16T03:10:06.655 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:532ce135:::161:head 2024-09-16T03:10:06.655 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:c78863e6:::162:head 2024-09-16T03:10:42.977 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:7462ddf6::io.RoundTripAppendPP (3013 ms) 2024-09-16T03:10:42.978 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-16T03:10:42.978 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2956 ms) 2024-09-16T03:10:42.978 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-16T03:10:42.978 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3019 ms) 2024-09-16T03:10:42.978 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-16T03:10:42.978 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3008 ms) 2024-09-16T03:10:42.978 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: setting pool EIO 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: max_success 95, min_failed 96 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3980 ms) 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3028 ms) 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (140867 ms total) 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2976 ms) 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9036 ms) 2024-09-16T03:10:42.979 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14978 ms) 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3008 ms) 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29998 ms total) 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3007 ms) 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3007 ms total) 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-16T03:10:42.980 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (27892 ms) 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6715 ms) 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6998 ms) 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7003 ms) 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2991 ms) 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7020 ms) 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7015 ms) 2024-09-16T03:10:42.981 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6971 ms) 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7115 ms) 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6935 ms) 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6973 ms) 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6985 ms) 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-16T03:10:42.982 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7019 ms) 2024-09-16T03:10:48.858 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:10:48.858 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:10:48.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:10:48.862 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:10:48.862 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:10:48.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:10:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 10 monclient: get_monmap_and_config 2024-09-16T03:10:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 10 monclient: build_initial_monmap 2024-09-16T03:10:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 10 monclient: monmap: 2024-09-16T03:10:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:48.995027+0000 2024-09-16T03:10:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:48.995027+0000 2024-09-16T03:10:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:48.996 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:48.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 1 Processor -- start 2024-09-16T03:10:48.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 1 -- start start 2024-09-16T03:10:48.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 10 monclient: init 2024-09-16T03:10:48.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:48.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:10:48.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 10 monclient(hunting): picked mon.noname-a con 0x7f51500a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.989+0000 7f51574db640 10 monclient(hunting): picked mon.noname-c con 0x7f51500a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.993+0000 7f51574db640 10 monclient(hunting): picked mon.noname-b con 0x7f5150073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.993+0000 7f51574db640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515009d9e0 con 0x7f51500a4d20 2024-09-16T03:10:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.993+0000 7f51574db640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515009dc20 con 0x7f51500a5810 2024-09-16T03:10:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.993+0000 7f51574db640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5150073360 con 0x7f5150073d40 2024-09-16T03:10:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.993+0000 7f51574db640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.993+0000 7f51574db640 10 monclient(hunting): authenticate will time out at 1339.644073s 2024-09-16T03:10:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.993+0000 7f5155a78640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5150073d40 0x7f5150140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:56220/0 (socket says 172.21.15.45:56220) 2024-09-16T03:10:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:48.993+0000 7f5155a78640 1 -- 172.21.15.45:0/2320599912 learned_addr learned my addr 172.21.15.45:0/2320599912 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.033+0000 7f5147fff640 1 -- 172.21.15.45:0/2320599912 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1071364513 0 0) 0x7f515009d9e0 con 0x7f51500a4d20 2024-09-16T03:10:49.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.033+0000 7f5147fff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.033+0000 7f5147fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.033+0000 7f5147fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.033+0000 7f5147fff640 10 monclient(hunting): my global_id is 104304 2024-09-16T03:10:49.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.033+0000 7f5147fff640 1 -- 172.21.15.45:0/2320599912 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f513c003530 con 0x7f51500a4d20 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 1 -- 172.21.15.45:0/2320599912 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 452194693 0 0) 0x7f513c003530 con 0x7f51500a4d20 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 1 -- 172.21.15.45:0/2320599912 >> v1:172.21.15.45:6790/0 conn(0x7f5150073d40 legacy=0x7f5150140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 1 -- 172.21.15.45:0/2320599912 >> v1:172.21.15.46:6789/0 conn(0x7f51500a5810 legacy=0x7f515013cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 1 monclient: found mon.noname-a 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 1 -- 172.21.15.45:0/2320599912 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515009cf30 con 0x7f51500a4d20 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 1 -- 172.21.15.45:0/2320599912 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f514c0016a0 con 0x7f51500a4d20 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: dump: 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 5 monclient: authenticate success, global_id 104304 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 1 -- 172.21.15.45:0/2320599912 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f514c001840 con 0x7f51500a4d20 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 1 -- 172.21.15.45:0/2320599912 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f514c001c90 con 0x7f51500a4d20 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5147fff640 10 monclient: dump: 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 4 set_mon_vals no callback set 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient: shutdown 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 -- 172.21.15.45:0/2320599912 >> v1:172.21.15.45:6789/0 conn(0x7f51500a4d20 legacy=0x7f51500a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 -- 172.21.15.45:0/2320599912 shutdown_connections 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 -- 172.21.15.45:0/2320599912 >> 172.21.15.45:0/2320599912 conn(0x7f51500aa720 msgr2=0x7f51500aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 -- 172.21.15.45:0/2320599912 shutdown_connections 2024-09-16T03:10:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 -- 172.21.15.45:0/2320599912 wait complete. 2024-09-16T03:10:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient: build_initial_monmap 2024-09-16T03:10:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient: monmap: 2024-09-16T03:10:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 librados: starting msgr at 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 librados: starting objecter 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 Processor -- start 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 -- start start 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 librados: setting wanted keys 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 librados: calling monclient init 2024-09-16T03:10:49.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient: init 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient(hunting): picked mon.noname-a con 0x7f5150073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient(hunting): picked mon.noname-c con 0x7f51500a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient(hunting): picked mon.noname-b con 0x7f51500a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5150140370 con 0x7f5150073d40 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5150140650 con 0x7f51500a5810 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51500a3bf0 con 0x7f51500a4d20 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f51574db640 10 monclient(hunting): authenticate will time out at 1339.691365s 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5155277640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5150073d40 0x7f51501cd1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33502/0 (socket says 172.21.15.45:33502) 2024-09-16T03:10:49.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5155277640 1 -- 172.21.15.45:0/1318665016 learned_addr learned my addr 172.21.15.45:0/1318665016 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 client.?.objecter ms_handle_connect 0x7f5150073d40 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 client.?.objecter ms_handle_connect 0x7f51500a4d20 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2451595698 0 0) 0x7f5150140370 con 0x7f5150073d40 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 monclient(hunting): my global_id is 104307 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5134004ef0 con 0x7f5150073d40 2024-09-16T03:10:49.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 client.?.objecter ms_handle_connect 0x7f51500a5810 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.037+0000 7f5145ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1436140059 0 0) 0x7f5134004ef0 con 0x7f5150073d40 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5150140370 con 0x7f5150073d40 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f514c001c50 con 0x7f5150073d40 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient(hunting): dump: 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.053 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 380985177 0 0) 0x7f5150140370 con 0x7f5150073d40 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 >> v1:172.21.15.45:6790/0 conn(0x7f51500a4d20 legacy=0x7f51501d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 >> v1:172.21.15.46:6789/0 conn(0x7f51500a5810 legacy=0x7f51501d5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 monclient: found mon.a 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51500a1e80 con 0x7f5150073d40 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 5 monclient: authenticate success, global_id 104307 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 10 monclient: _renew_subs 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5150140090 con 0x7f5150073d40 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 10 client.104307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 10 monclient: _renew_subs 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f515013db20 con 0x7f5150073d40 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 1 librados: init done 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f514c0017c0 con 0x7f5150073d40 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f514c003b10 con 0x7f5150073d40 2024-09-16T03:10:49.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51477fe640 4 set_mon_vals no callback set 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 monclient: dump: 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.056 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f514c003cb0 con 0x7f5150073d40 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5130005090 con 0x7f5150073d40 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(276..276 src has 1..276) ==== 6197+0+0 (unknown 943674340 0 0) 0x7f514c04baa0 con 0x7f5150073d40 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 client.104307.objecter ms_dispatch 0x7f5150000ba0 osd_map(276..276 src has 1..276) 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 3 client.104307.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 3 client.104307.objecter handle_osd_map decoding full epoch 276 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 20 client.104307.objecter dump_active .. 0 homeless 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5154a76640 10 monclient: get_auth_request con 0x7f513403e0d0 auth_method 0 2024-09-16T03:10:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.041+0000 7f5145ffb640 10 client.104307.objecter ms_handle_connect 0x7f513403e0d0 2024-09-16T03:10:49.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.045+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f514c016b10 con 0x7f5150073d40 2024-09-16T03:10:49.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.045+0000 7f5145ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.045+0000 7f5145ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.281+0000 7f51574db640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:10:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.281+0000 7f51574db640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:10:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.281+0000 7f51574db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.281+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5130005380 con 0x7f5150073d40 2024-09-16T03:10:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f5145ffb640 1 -- 172.21.15.45:0/1318665016 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 1840295534) 0x7f514c016410 con 0x7f5150073d40 2024-09-16T03:10:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f5145ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:10:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f5145ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:49.289 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:10:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 10 librados: watch_flush enter 2024-09-16T03:10:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 10 librados: watch_flush exit 2024-09-16T03:10:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 20 client.104307.objecter shutdown clearing up homeless session... 2024-09-16T03:10:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 10 client.104307.objecter successfully canceled tick 2024-09-16T03:10:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 >> v1:172.21.15.46:6812/149360280 conn(0x7f513403e0d0 legacy=0x7f5134040590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 10 monclient: shutdown 2024-09-16T03:10:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 >> v1:172.21.15.45:6789/0 conn(0x7f5150073d40 legacy=0x7f51501cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 shutdown_connections 2024-09-16T03:10:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 >> 172.21.15.45:0/1318665016 conn(0x7f51500aa720 msgr2=0x7f51500a74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 shutdown_connections 2024-09-16T03:10:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 1 -- 172.21.15.45:0/1318665016 wait complete. 2024-09-16T03:10:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.285+0000 7f51574db640 1 librados: shutdown 2024-09-16T03:10:49.307 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:10:49.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:10:49.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:10:49.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient: get_monmap_and_config 2024-09-16T03:10:49.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient: build_initial_monmap 2024-09-16T03:10:49.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient: monmap: 2024-09-16T03:10:49.426 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:49.426 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.426 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:49.424890+0000 2024-09-16T03:10:49.426 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:49.424890+0000 2024-09-16T03:10:49.426 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:49.426 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.426 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:49.427 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:49.427 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:49.427 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 Processor -- start 2024-09-16T03:10:49.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 -- start start 2024-09-16T03:10:49.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient: init 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient(hunting): picked mon.noname-b con 0x7f8d80137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient(hunting): picked mon.noname-a con 0x7f8d80138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f8d8013e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d800a4f90 con 0x7f8d80138100 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d800a4cb0 con 0x7f8d8013e2a0 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8009dc70 con 0x7f8d80137d20 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient(hunting): authenticate will time out at 1340.074332s 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8d80138100 0x7f8d8013cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33514/0 (socket says 172.21.15.45:33514) 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7ffff640 1 -- 172.21.15.45:0/154844754 learned_addr learned my addr 172.21.15.45:0/154844754 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 521618712 0 0) 0x7f8d800a4f90 con 0x7f8d80138100 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient(hunting): my global_id is 104310 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d68003530 con 0x7f8d80138100 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2045604821 0 0) 0x7f8d8009dc70 con 0x7f8d80137d20 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient(hunting): my global_id is 114101 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d800a4f90 con 0x7f8d80137d20 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1788394747 0 0) 0x7f8d800a4f90 con 0x7f8d80137d20 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 >> v1:172.21.15.46:6789/0 conn(0x7f8d8013e2a0 legacy=0x7f8d80140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 >> v1:172.21.15.45:6789/0 conn(0x7f8d80138100 legacy=0x7f8d8013cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 monclient: found mon.noname-b 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8009cc50 con 0x7f8d80137d20 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 5 monclient: authenticate success, global_id 114101 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8d70001680 con 0x7f8d80137d20 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: dump: 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.440 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d700018a0 con 0x7f8d80137d20 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 1 -- 172.21.15.45:0/154844754 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8d70001cb0 con 0x7f8d80137d20 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 4 set_mon_vals no callback set 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:49.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d7f7fe640 10 monclient: dump: 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 10 monclient: shutdown 2024-09-16T03:10:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 -- 172.21.15.45:0/154844754 >> v1:172.21.15.45:6790/0 conn(0x7f8d80137d20 legacy=0x7f8d800a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 -- 172.21.15.45:0/154844754 shutdown_connections 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 -- 172.21.15.45:0/154844754 >> 172.21.15.45:0/154844754 conn(0x7f8d800aa110 msgr2=0x7f8d800aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 -- 172.21.15.45:0/154844754 shutdown_connections 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.421+0000 7f8d86ae6640 1 -- 172.21.15.45:0/154844754 wait complete. 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient: build_initial_monmap 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient: monmap: 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 librados: starting msgr at 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 librados: starting objecter 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 Processor -- start 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 -- start start 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 librados: setting wanted keys 2024-09-16T03:10:49.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 librados: calling monclient init 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient: init 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f8d80137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient(hunting): picked mon.noname-b con 0x7f8d80138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient(hunting): picked mon.noname-a con 0x7f8d8013e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d80141b80 con 0x7f8d8013e2a0 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d800a1300 con 0x7f8d80138100 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d80139b20 con 0x7f8d80137d20 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient(hunting): authenticate will time out at 1340.077629s 2024-09-16T03:10:49.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d84882640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8d80137d20 0x7f8d80138950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:56256/0 (socket says 172.21.15.45:56256) 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d84882640 1 -- 172.21.15.45:0/3608475698 learned_addr learned my addr 172.21.15.45:0/3608475698 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 client.?.objecter ms_handle_connect 0x7f8d8013e2a0 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 client.?.objecter ms_handle_connect 0x7f8d80137d20 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2203851107 0 0) 0x7f8d80139b20 con 0x7f8d80137d20 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): my global_id is 114104 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d6c004ec0 con 0x7f8d80137d20 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2981007278 0 0) 0x7f8d80141b80 con 0x7f8d8013e2a0 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): my global_id is 104313 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d80139b20 con 0x7f8d8013e2a0 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 client.?.objecter ms_handle_connect 0x7f8d80138100 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 873945837 0 0) 0x7f8d80139b20 con 0x7f8d8013e2a0 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d80141b80 con 0x7f8d8013e2a0 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8d78001c00 con 0x7f8d8013e2a0 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): dump: 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.447 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1555157956 0 0) 0x7f8d6c004ec0 con 0x7f8d80137d20 2024-09-16T03:10:49.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d80139b20 con 0x7f8d80137d20 2024-09-16T03:10:49.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8d70001c70 con 0x7f8d80137d20 2024-09-16T03:10:49.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:49.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:49.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): dump: 2024-09-16T03:10:49.449 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.449 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.449 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.449 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.449 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341986253 0 0) 0x7f8d800a1300 con 0x7f8d80138100 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient(hunting): my global_id is 22810 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d6c004ec0 con 0x7f8d80138100 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3443611354 0 0) 0x7f8d80141b80 con 0x7f8d8013e2a0 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 >> v1:172.21.15.45:6790/0 conn(0x7f8d80137d20 legacy=0x7f8d80138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 >> v1:172.21.15.46:6789/0 conn(0x7f8d80138100 legacy=0x7f8d80139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 monclient: found mon.a 2024-09-16T03:10:49.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8013be80 con 0x7f8d8013e2a0 2024-09-16T03:10:49.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 5 monclient: authenticate success, global_id 104313 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient: _renew_subs 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8013b5e0 con 0x7f8d8013e2a0 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 client.104313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient: _renew_subs 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8d8013a8e0 con 0x7f8d8013e2a0 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 librados: init done 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d78003980 con 0x7f8d8013e2a0 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8d78003d10 con 0x7f8d8013e2a0 2024-09-16T03:10:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 monclient: dump: 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.454 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7effd640 4 set_mon_vals no callback set 2024-09-16T03:10:49.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f8d78003f70 con 0x7f8d8013e2a0 2024-09-16T03:10:49.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(276..276 src has 1..276) ==== 6197+0+0 (unknown 943674340 0 0) 0x7f8d7804ab70 con 0x7f8d8013e2a0 2024-09-16T03:10:49.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 10 client.104313.objecter ms_dispatch 0x7f8d80000ba0 osd_map(276..276 src has 1..276) 2024-09-16T03:10:49.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 3 client.104313.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T03:10:49.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 3 client.104313.objecter handle_osd_map decoding full epoch 276 2024-09-16T03:10:49.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8d800008e0 con 0x7f8d8009e910 2024-09-16T03:10:49.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.425+0000 7f8d7d7fa640 20 client.104313.objecter dump_active .. 0 homeless 2024-09-16T03:10:49.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 client.104313.objecter ms_handle_connect 0x7f8d8009e910 2024-09-16T03:10:49.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 client.104313.objecter resend_mon_ops 2024-09-16T03:10:49.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d84882640 10 monclient: get_auth_request con 0x7f8d6c03e290 auth_method 0 2024-09-16T03:10:49.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 883738219 0 0) 0x7f8d800008e0 con 0x7f8d8009e910 2024-09-16T03:10:49.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 monclient(hunting): my global_id is 104316 2024-09-16T03:10:49.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d80141b80 con 0x7f8d8009e910 2024-09-16T03:10:49.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2040350044 0 0) 0x7f8d80141b80 con 0x7f8d8009e910 2024-09-16T03:10:49.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d800008e0 con 0x7f8d8009e910 2024-09-16T03:10:49.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8d74001290 con 0x7f8d8009e910 2024-09-16T03:10:49.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 monclient: dump: 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.457 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 client.104313.objecter ms_handle_connect 0x7f8d6c03e290 2024-09-16T03:10:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1281580786 0 0) 0x7f8d800008e0 con 0x7f8d8009e910 2024-09-16T03:10:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d800770b0 con 0x7f8d8009e910 2024-09-16T03:10:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8d740013a0 con 0x7f8d8009e910 2024-09-16T03:10:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 >> v1:172.21.15.45:6789/0 conn(0x7f8d8009e910 legacy=0x7f8d8009ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.429+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 >> v1:172.21.15.45:6789/0 conn(0x7f8d8009e910 legacy=0x7f8d8009ecf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d86ae6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:10:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8d8009f810 con 0x7f8d8009f400 2024-09-16T03:10:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 client.104313.objecter ms_handle_connect 0x7f8d8009f400 2024-09-16T03:10:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 client.104313.objecter resend_mon_ops 2024-09-16T03:10:49.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895858645 0 0) 0x7f8d8009f810 con 0x7f8d8009f400 2024-09-16T03:10:49.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 monclient(hunting): my global_id is 104319 2024-09-16T03:10:49.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d800008e0 con 0x7f8d8009f400 2024-09-16T03:10:49.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1551562409 0 0) 0x7f8d800008e0 con 0x7f8d8009f400 2024-09-16T03:10:49.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d8009f810 con 0x7f8d8009f400 2024-09-16T03:10:49.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8d74004990 con 0x7f8d8009f400 2024-09-16T03:10:49.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 monclient: dump: 2024-09-16T03:10:49.462 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2362502722 0 0) 0x7f8d8009f810 con 0x7f8d8009f400 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d8013d5e0 con 0x7f8d8009f400 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4137 (unknown 0 0 998073344) 0x7f8d74005e60 con 0x7f8d8009f400 2024-09-16T03:10:49.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:10:49.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.433+0000 7f8d7d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:49.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 >> v1:172.21.15.45:6789/0 conn(0x7f8d8009f400 legacy=0x7f8d801da4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d7d7fa640 1 -- 172.21.15.45:0/3608475698 >> v1:172.21.15.45:6789/0 conn(0x7f8d8009f400 legacy=0x7f8d801da4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:49.464 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:10:49.464 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:10:49.464 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:10:49.464 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:10:49.465 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 26, 2024-09-16T03:10:49.465 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 108534, 2024-09-16T03:10:49.465 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:10:49.465 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:10:49.465 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:10:49.465 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:10:49.466 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.466 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:10:49.466 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:49.466 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:10:49.466 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:10:49.466 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:49.466 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:49.466 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:49.466 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:10:49.467 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:10:49.469 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:49.470 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:49.471 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:10:49.472 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:49.473 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.474 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.475 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:10:49.476 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:10:49.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.476 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:49.476 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:49.476 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:49.476 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.476 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.476 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:10:49.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.477 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:49.477 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:49.477 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:49.477 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.477 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:10:49.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.478 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:49.478 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:49.478 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:10:49.478 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.478 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.479 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:10:49.479 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:10:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 10 librados: watch_flush enter 2024-09-16T03:10:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 10 librados: watch_flush exit 2024-09-16T03:10:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 20 client.104313.objecter shutdown clearing up homeless session... 2024-09-16T03:10:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 10 client.104313.objecter successfully canceled tick 2024-09-16T03:10:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 >> v1:172.21.15.46:6812/149360280 conn(0x7f8d6c03e290 legacy=0x7f8d6c040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 10 monclient: shutdown 2024-09-16T03:10:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 >> v1:172.21.15.45:6789/0 conn(0x7f8d8013e2a0 legacy=0x7f8d801d4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d85083640 1 -- 172.21.15.45:0/3608475698 reap_dead start 2024-09-16T03:10:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d85083640 1 -- 172.21.15.45:0/3608475698 reap_dead start 2024-09-16T03:10:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 shutdown_connections 2024-09-16T03:10:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 >> 172.21.15.45:0/3608475698 conn(0x7f8d800aa110 msgr2=0x7f8d800742a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:49.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 shutdown_connections 2024-09-16T03:10:49.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 1 -- 172.21.15.45:0/3608475698 wait complete. 2024-09-16T03:10:49.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.437+0000 7f8d86ae6640 1 librados: shutdown 2024-09-16T03:10:49.484 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:10:49.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:10:49.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:10:49.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.601+0000 7f7e8ba4d640 10 monclient: get_monmap_and_config 2024-09-16T03:10:49.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.601+0000 7f7e8ba4d640 10 monclient: build_initial_monmap 2024-09-16T03:10:49.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.601+0000 7f7e8ba4d640 10 monclient: monmap: 2024-09-16T03:10:49.608 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:49.608 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.608 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:49.607867+0000 2024-09-16T03:10:49.608 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:49.607867+0000 2024-09-16T03:10:49.609 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:49.609 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.609 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:49.609 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:49.609 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:49.609 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 Processor -- start 2024-09-16T03:10:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 -- start start 2024-09-16T03:10:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient: init 2024-09-16T03:10:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:10:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e84137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e84138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e8413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e840a4f90 con 0x7f7e84137d20 2024-09-16T03:10:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e840a4cb0 con 0x7f7e8413e2a0 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e8409dc70 con 0x7f7e84138100 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient(hunting): authenticate will time out at 1340.257687s 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e897e9640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7e84137d20 0x7f7e840a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33554/0 (socket says 172.21.15.45:33554) 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e897e9640 1 -- 172.21.15.45:0/693166551 learned_addr learned my addr 172.21.15.45:0/693166551 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4203746268 0 0) 0x7f7e840a4f90 con 0x7f7e84137d20 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient(hunting): my global_id is 104322 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e70003530 con 0x7f7e84137d20 2024-09-16T03:10:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2770596649 0 0) 0x7f7e8409dc70 con 0x7f7e84138100 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient(hunting): my global_id is 114107 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e840a4f90 con 0x7f7e84138100 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4280291704 0 0) 0x7f7e70003530 con 0x7f7e84137d20 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 >> v1:172.21.15.45:6790/0 conn(0x7f7e84138100 legacy=0x7f7e8413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 >> v1:172.21.15.46:6789/0 conn(0x7f7e8413e2a0 legacy=0x7f7e84140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 monclient: found mon.noname-a 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8409cc50 con 0x7f7e84137d20 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 5 monclient: authenticate success, global_id 104322 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7e800016a0 con 0x7f7e84137d20 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: dump: 2024-09-16T03:10:49.622 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.623 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.623 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.623 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.623 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.623 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.623 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.623 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e80001840 con 0x7f7e84137d20 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 1 -- 172.21.15.45:0/693166551 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7e80001c90 con 0x7f7e84137d20 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 4 set_mon_vals no callback set 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e7bfff640 10 monclient: dump: 2024-09-16T03:10:49.624 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 10 monclient: shutdown 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/693166551 >> v1:172.21.15.45:6789/0 conn(0x7f7e84137d20 legacy=0x7f7e840a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/693166551 shutdown_connections 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/693166551 >> 172.21.15.45:0/693166551 conn(0x7f7e840aa110 msgr2=0x7f7e840aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/693166551 shutdown_connections 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.605+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/693166551 wait complete. 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient: build_initial_monmap 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient: monmap: 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:49.627 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:49.627 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:49.627 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:49.627 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 librados: starting msgr at 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 librados: starting objecter 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 Processor -- start 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 -- start start 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 librados: setting wanted keys 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 librados: calling monclient init 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient: init 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:49.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e84137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e84138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e8413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e84139740 con 0x7f7e84137d20 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e8413b790 con 0x7f7e8413e2a0 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e8413ba70 con 0x7f7e84138100 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient(hunting): authenticate will time out at 1340.260991s 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e897e9640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7e84137d20 0x7f7e84138920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33562/0 (socket says 172.21.15.45:33562) 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e897e9640 1 -- 172.21.15.45:0/3799102797 learned_addr learned my addr 172.21.15.45:0/3799102797 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 client.?.objecter ms_handle_connect 0x7f7e84137d20 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 client.?.objecter ms_handle_connect 0x7f7e84138100 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2029915268 0 0) 0x7f7e84139740 con 0x7f7e84137d20 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): my global_id is 104325 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e68004ec0 con 0x7f7e84137d20 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606361204 0 0) 0x7f7e8413ba70 con 0x7f7e84138100 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): my global_id is 114110 2024-09-16T03:10:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e84139740 con 0x7f7e84138100 2024-09-16T03:10:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2651790782 0 0) 0x7f7e68004ec0 con 0x7f7e84137d20 2024-09-16T03:10:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e8413ba70 con 0x7f7e84137d20 2024-09-16T03:10:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 client.?.objecter ms_handle_connect 0x7f7e8413e2a0 2024-09-16T03:10:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7e80001b90 con 0x7f7e84137d20 2024-09-16T03:10:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): dump: 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 133126299 0 0) 0x7f7e84139740 con 0x7f7e84138100 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e68004ec0 con 0x7f7e84138100 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7e740018b0 con 0x7f7e84138100 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): dump: 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.633 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3007613358 0 0) 0x7f7e8413ba70 con 0x7f7e84137d20 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 >> v1:172.21.15.45:6790/0 conn(0x7f7e84138100 legacy=0x7f7e84139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 >> v1:172.21.15.46:6789/0 conn(0x7f7e8413e2a0 legacy=0x7f7e841d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 monclient: found mon.a 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8413aa90 con 0x7f7e84137d20 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 5 monclient: authenticate success, global_id 104325 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient: _renew_subs 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8413d790 con 0x7f7e84137d20 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 client.104325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient: _renew_subs 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e8413c620 con 0x7f7e84137d20 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 librados: init done 2024-09-16T03:10:49.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e800017a0 con 0x7f7e84137d20 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7e80004da0 con 0x7f7e84137d20 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: dump: 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.636 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e7b7fe640 4 set_mon_vals no callback set 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7e640065d0 con 0x7f7e64002870 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 client.104325.objecter ms_handle_connect 0x7f7e64002870 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 client.104325.objecter resend_mon_ops 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7e80010e10 con 0x7f7e84137d20 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(276..276 src has 1..276) ==== 6197+0+0 (unknown 943674340 0 0) 0x7f7e8004ba90 con 0x7f7e84137d20 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 client.104325.objecter ms_dispatch 0x7f7e84000ba0 osd_map(276..276 src has 1..276) 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 3 client.104325.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 3 client.104325.objecter handle_osd_map decoding full epoch 276 2024-09-16T03:10:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 20 client.104325.objecter dump_active .. 0 homeless 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955235044 0 0) 0x7f7e640065d0 con 0x7f7e64002870 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient(hunting): my global_id is 114113 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e8413ba70 con 0x7f7e64002870 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3103041823 0 0) 0x7f7e8413ba70 con 0x7f7e64002870 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e640065d0 con 0x7f7e64002870 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7e740037c0 con 0x7f7e64002870 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e89fea640 10 monclient: get_auth_request con 0x7f7e6803e040 auth_method 0 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 10 monclient: dump: 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1600807168 0 0) 0x7f7e640065d0 con 0x7f7e64002870 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.609+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e6400a170 con 0x7f7e64002870 2024-09-16T03:10:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.613+0000 7f7e79ffb640 10 client.104325.objecter ms_handle_connect 0x7f7e6803e040 2024-09-16T03:10:49.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.613+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7e740059e0 con 0x7f7e64002870 2024-09-16T03:10:49.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.613+0000 7f7e79ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.613+0000 7f7e79ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:49.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.613+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 >> v1:172.21.15.45:6790/0 conn(0x7f7e64002870 legacy=0x7f7e64004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.613+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 >> v1:172.21.15.45:6790/0 conn(0x7f7e64002870 legacy=0x7f7e64004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7e64009e30 con 0x7f7e640090f0 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 client.104325.objecter ms_handle_connect 0x7f7e640090f0 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 client.104325.objecter resend_mon_ops 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1839263337 0 0) 0x7f7e64009e30 con 0x7f7e640090f0 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 monclient(hunting): my global_id is 114116 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e640065d0 con 0x7f7e640090f0 2024-09-16T03:10:49.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4269536545 0 0) 0x7f7e640065d0 con 0x7f7e640090f0 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e64009e30 con 0x7f7e640090f0 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7e74005170 con 0x7f7e640090f0 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 monclient: dump: 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.643 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1065118123 0 0) 0x7f7e64009e30 con 0x7f7e640090f0 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e6400c400 con 0x7f7e640090f0 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3965 (unknown 0 0 4162236538) 0x7f7e74004f90 con 0x7f7e640090f0 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 >> v1:172.21.15.45:6790/0 conn(0x7f7e640090f0 legacy=0x7f7e64009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e79ffb640 1 -- 172.21.15.45:0/3799102797 >> v1:172.21.15.45:6790/0 conn(0x7f7e640090f0 legacy=0x7f7e64009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:49.645 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:10:49.645 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:10:49.645 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:10:49.645 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:10:49.645 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 26, 2024-09-16T03:10:49.645 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 679, 2024-09-16T03:10:49.645 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:10:49.645 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:10:49.645 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:10:49.646 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:10:49.646 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.646 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:10:49.646 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:49.646 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:10:49.646 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:10:49.646 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.647 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:49.648 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:10:49.649 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:49.650 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:10:49.651 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:49.652 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.653 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.654 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:49.656 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:49.657 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:49.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.657 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.657 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:10:49.657 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:10:49.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 10 librados: watch_flush enter 2024-09-16T03:10:49.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 10 librados: watch_flush exit 2024-09-16T03:10:49.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 20 client.104325.objecter shutdown clearing up homeless session... 2024-09-16T03:10:49.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 10 client.104325.objecter successfully canceled tick 2024-09-16T03:10:49.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 >> v1:172.21.15.46:6812/149360280 conn(0x7f7e6803e040 legacy=0x7f7e68040500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 10 monclient: shutdown 2024-09-16T03:10:49.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:49.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 >> v1:172.21.15.45:6789/0 conn(0x7f7e84137d20 legacy=0x7f7e84138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e89fea640 1 -- 172.21.15.45:0/3799102797 reap_dead start 2024-09-16T03:10:49.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e89fea640 1 -- 172.21.15.45:0/3799102797 reap_dead start 2024-09-16T03:10:49.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 shutdown_connections 2024-09-16T03:10:49.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 >> 172.21.15.45:0/3799102797 conn(0x7f7e840aa110 msgr2=0x7f7e840a07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:49.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 shutdown_connections 2024-09-16T03:10:49.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 1 -- 172.21.15.45:0/3799102797 wait complete. 2024-09-16T03:10:49.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.617+0000 7f7e8ba4d640 1 librados: shutdown 2024-09-16T03:10:49.660 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:10:49.660 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:10:49.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:10:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.777+0000 7f57e1943640 10 monclient: get_monmap_and_config 2024-09-16T03:10:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.777+0000 7f57e1943640 10 monclient: build_initial_monmap 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.777+0000 7f57e1943640 10 monclient: monmap: 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:49.783355+0000 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:49.783355+0000 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:49.785 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 Processor -- start 2024-09-16T03:10:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- start start 2024-09-16T03:10:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient: init 2024-09-16T03:10:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:10:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient(hunting): picked mon.noname-b con 0x7f57dc135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient(hunting): picked mon.noname-a con 0x7f57dc09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient(hunting): picked mon.noname-c con 0x7f57dc13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57dc09f7e0 con 0x7f57dc09fc40 2024-09-16T03:10:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57dc09b160 con 0x7f57dc13c010 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57dc09b440 con 0x7f57dc135950 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient(hunting): authenticate will time out at 1340.433266s 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57daffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f57dc135950 0x7f57dc135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:56292/0 (socket says 172.21.15.45:56292) 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57daffd640 1 -- 172.21.15.45:0/158423874 learned_addr learned my addr 172.21.15.45:0/158423874 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57d27fc640 1 --1- 172.21.15.45:0/158423874 >> v1:172.21.15.45:6789/0 conn(0x7f57dc09fc40 0x7f57dc13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33572/0 (socket says 172.21.15.45:33572) 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132685750 0 0) 0x7f57dc09b440 con 0x7f57dc135950 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient(hunting): my global_id is 114119 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57c8003530 con 0x7f57dc135950 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1649219016 0 0) 0x7f57dc09f7e0 con 0x7f57dc09fc40 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient(hunting): my global_id is 104328 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57dc09b440 con 0x7f57dc09fc40 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1188172596 0 0) 0x7f57dc09b440 con 0x7f57dc09fc40 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 >> v1:172.21.15.45:6790/0 conn(0x7f57dc135950 legacy=0x7f57dc135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 >> v1:172.21.15.46:6789/0 conn(0x7f57dc13c010 legacy=0x7f57dc13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 monclient: found mon.noname-a 2024-09-16T03:10:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57dc136440 con 0x7f57dc09fc40 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f57c4001680 con 0x7f57dc09fc40 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 5 monclient: authenticate success, global_id 104328 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: dump: 2024-09-16T03:10:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.792 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f57c40018a0 con 0x7f57dc09fc40 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 1 -- 172.21.15.45:0/158423874 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f57c4001d30 con 0x7f57dc09fc40 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57da7fc640 10 monclient: dump: 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 4 set_mon_vals no callback set 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient: shutdown 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- 172.21.15.45:0/158423874 >> v1:172.21.15.45:6789/0 conn(0x7f57dc09fc40 legacy=0x7f57dc13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- 172.21.15.45:0/158423874 shutdown_connections 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- 172.21.15.45:0/158423874 >> 172.21.15.45:0/158423874 conn(0x7f57dc0a2bd0 msgr2=0x7f57dc0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- 172.21.15.45:0/158423874 shutdown_connections 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- 172.21.15.45:0/158423874 wait complete. 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient: build_initial_monmap 2024-09-16T03:10:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient: monmap: 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 librados: starting msgr at 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 librados: starting objecter 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 Processor -- start 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 -- start start 2024-09-16T03:10:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 librados: setting wanted keys 2024-09-16T03:10:49.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 1 librados: calling monclient init 2024-09-16T03:10:49.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.781+0000 7f57e1943640 10 monclient: init 2024-09-16T03:10:49.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:49.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:10:49.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient(hunting): picked mon.noname-a con 0x7f57dc09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:49.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient(hunting): picked mon.noname-c con 0x7f57dc135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:49.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient(hunting): picked mon.noname-b con 0x7f57dc13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:49.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57dc071ad0 con 0x7f57dc09fc40 2024-09-16T03:10:49.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57dc06f1d0 con 0x7f57dc13c010 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57dc071fd0 con 0x7f57dc135950 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient(hunting): authenticate will time out at 1340.436467s 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57daffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f57dc09fc40 0x7f57dc06e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33580/0 (socket says 172.21.15.45:33580) 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57daffd640 1 -- 172.21.15.45:0/3915986715 learned_addr learned my addr 172.21.15.45:0/3915986715 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.?.objecter ms_handle_connect 0x7f57dc135950 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.?.objecter ms_handle_connect 0x7f57dc09fc40 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587398023 0 0) 0x7f57dc071fd0 con 0x7f57dc135950 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): my global_id is 114122 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57bc004ec0 con 0x7f57dc135950 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2803946273 0 0) 0x7f57dc071ad0 con 0x7f57dc09fc40 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): my global_id is 104331 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57dc071fd0 con 0x7f57dc09fc40 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.?.objecter ms_handle_connect 0x7f57dc13c010 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3105499480 0 0) 0x7f57dc071fd0 con 0x7f57dc09fc40 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57dc071ad0 con 0x7f57dc09fc40 2024-09-16T03:10:49.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4018339044 0 0) 0x7f57bc004ec0 con 0x7f57dc135950 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57dc071fd0 con 0x7f57dc135950 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f57cc0016a0 con 0x7f57dc09fc40 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): dump: 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.805 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f57c4001bf0 con 0x7f57dc135950 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): dump: 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932098464 0 0) 0x7f57dc06f1d0 con 0x7f57dc13c010 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): my global_id is 22822 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57bc004ec0 con 0x7f57dc13c010 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 917662010 0 0) 0x7f57dc071ad0 con 0x7f57dc09fc40 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 >> v1:172.21.15.45:6790/0 conn(0x7f57dc135950 legacy=0x7f57dc06eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 >> v1:172.21.15.46:6789/0 conn(0x7f57dc13c010 legacy=0x7f57dc1d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 monclient: found mon.a 2024-09-16T03:10:49.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57dc071230 con 0x7f57dc09fc40 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 5 monclient: authenticate success, global_id 104331 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient: _renew_subs 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f57dc071510 con 0x7f57dc09fc40 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 client.104331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient: _renew_subs 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:49.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f57dc06f8a0 con 0x7f57dc09fc40 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 1 librados: init done 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f57cc004b70 con 0x7f57dc09fc40 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f57cc004ec0 con 0x7f57dc09fc40 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d9ffb640 4 set_mon_vals no callback set 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: dump: 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.809 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f57c00065d0 con 0x7f57c0002870 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f57cc010f30 con 0x7f57dc09fc40 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(276..276 src has 1..276) ==== 6197+0+0 (unknown 943674340 0 0) 0x7f57cc04be00 con 0x7f57dc09fc40 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.104331.objecter ms_dispatch 0x7f57dc000ba0 osd_map(276..276 src has 1..276) 2024-09-16T03:10:49.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 3 client.104331.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 3 client.104331.objecter handle_osd_map decoding full epoch 276 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 20 client.104331.objecter dump_active .. 0 homeless 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.104331.objecter ms_handle_connect 0x7f57c0002870 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.104331.objecter resend_mon_ops 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3549999080 0 0) 0x7f57c00065d0 con 0x7f57c0002870 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57db7fe640 10 monclient: get_auth_request con 0x7f57bc03e290 auth_method 0 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient(hunting): my global_id is 22825 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57dc071ad0 con 0x7f57c0002870 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 client.104331.objecter ms_handle_connect 0x7f57bc03e290 2024-09-16T03:10:49.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1320763808 0 0) 0x7f57dc071ad0 con 0x7f57c0002870 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57c00065d0 con 0x7f57c0002870 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f57c4004990 con 0x7f57c0002870 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 10 monclient: dump: 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.812 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3900964314 0 0) 0x7f57c00065d0 con 0x7f57c0002870 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.785+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57c000a170 con 0x7f57c0002870 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.789+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f57c4006b90 con 0x7f57c0002870 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.789+0000 7f57d8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.789+0000 7f57d8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.789+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 >> v1:172.21.15.46:6789/0 conn(0x7f57c0002870 legacy=0x7f57c0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.789+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 >> v1:172.21.15.46:6789/0 conn(0x7f57c0002870 legacy=0x7f57c0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57e1943640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f57c0009e80 con 0x7f57c0009140 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57d8ff9640 10 client.104331.objecter ms_handle_connect 0x7f57c0009140 2024-09-16T03:10:49.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57d8ff9640 10 client.104331.objecter resend_mon_ops 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2140041799 0 0) 0x7f57c0009e80 con 0x7f57c0009140 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57d8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57d8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57d8ff9640 10 monclient(hunting): my global_id is 22828 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.793+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57c00065d0 con 0x7f57c0009140 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3019618998 0 0) 0x7f57c00065d0 con 0x7f57c0009140 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57c0009e80 con 0x7f57c0009140 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f57c40062a0 con 0x7f57c0009140 2024-09-16T03:10:49.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 10 monclient: dump: 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.816 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1914273028 0 0) 0x7f57c0009e80 con 0x7f57c0009140 2024-09-16T03:10:49.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f57c000c420 con 0x7f57c0009140 2024-09-16T03:10:49.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 1692925279) 0x7f57c4006040 con 0x7f57c0009140 2024-09-16T03:10:49.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:10:49.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:49.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 >> v1:172.21.15.46:6789/0 conn(0x7f57c0009140 legacy=0x7f57c00095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57d8ff9640 1 -- 172.21.15.45:0/3915986715 >> v1:172.21.15.46:6789/0 conn(0x7f57c0009140 legacy=0x7f57c00095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:49.816 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 26, 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 371603, 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:10:49.817 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.818 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.819 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:10:49.820 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:49.821 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:49.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.823 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:10:49.824 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:49.825 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:10:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:49.828 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:49.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:49.829 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:10:49.829 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:10:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 10 librados: watch_flush enter 2024-09-16T03:10:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 10 librados: watch_flush exit 2024-09-16T03:10:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 20 client.104331.objecter shutdown clearing up homeless session... 2024-09-16T03:10:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 10 client.104331.objecter successfully canceled tick 2024-09-16T03:10:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 >> v1:172.21.15.46:6812/149360280 conn(0x7f57bc03e290 legacy=0x7f57bc040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 10 monclient: shutdown 2024-09-16T03:10:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 >> v1:172.21.15.45:6789/0 conn(0x7f57dc09fc40 legacy=0x7f57dc06e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57db7fe640 1 -- 172.21.15.45:0/3915986715 reap_dead start 2024-09-16T03:10:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57db7fe640 1 -- 172.21.15.45:0/3915986715 reap_dead start 2024-09-16T03:10:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 shutdown_connections 2024-09-16T03:10:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 >> 172.21.15.45:0/3915986715 conn(0x7f57dc0a2bd0 msgr2=0x7f57dc13ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 shutdown_connections 2024-09-16T03:10:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 1 -- 172.21.15.45:0/3915986715 wait complete. 2024-09-16T03:10:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.797+0000 7f57e1943640 1 librados: shutdown 2024-09-16T03:10:49.833 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:10:49.833 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:10:49.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:10:49.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.941+0000 7fd227560640 10 monclient: get_monmap_and_config 2024-09-16T03:10:49.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.941+0000 7fd227560640 10 monclient: build_initial_monmap 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.941+0000 7fd227560640 10 monclient: monmap: 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:49.947496+0000 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:49.947496+0000 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:49.948 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 Processor -- start 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- start start 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: init 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): picked mon.noname-a con 0x7fd22009b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): picked mon.noname-c con 0x7fd22009b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): picked mon.noname-b con 0x7fd22013b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22009cc20 con 0x7fd22009b8e0 2024-09-16T03:10:49.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22009d820 con 0x7fd22009b250 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22009b6a0 con 0x7fd22013b000 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): authenticate will time out at 1340.596693s 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd2252fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd22009b8e0 0x7fd22009bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33594/0 (socket says 172.21.15.45:33594) 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd2252fc640 1 -- 172.21.15.45:0/2076945505 learned_addr learned my addr 172.21.15.45:0/2076945505 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1629484863 0 0) 0x7fd22009b6a0 con 0x7fd22013b000 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient(hunting): my global_id is 114125 2024-09-16T03:10:49.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd20c003530 con 0x7fd22013b000 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1545496437 0 0) 0x7fd22009cc20 con 0x7fd22009b8e0 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient(hunting): my global_id is 104334 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd22009b6a0 con 0x7fd22009b8e0 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1842626305 0 0) 0x7fd20c003530 con 0x7fd22013b000 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 >> v1:172.21.15.46:6789/0 conn(0x7fd22009b250 legacy=0x7fd2201398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 >> v1:172.21.15.45:6789/0 conn(0x7fd22009b8e0 legacy=0x7fd22009bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 monclient: found mon.noname-b 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2200a21f0 con 0x7fd22013b000 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd218001a20 con 0x7fd22013b000 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 5 monclient: authenticate success, global_id 114125 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: dump: 2024-09-16T03:10:49.958 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd218001bc0 con 0x7fd22013b000 2024-09-16T03:10:49.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 1 -- 172.21.15.45:0/2076945505 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd218001f90 con 0x7fd22013b000 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd217fff640 10 monclient: dump: 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 4 set_mon_vals no callback set 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: shutdown 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- 172.21.15.45:0/2076945505 >> v1:172.21.15.45:6790/0 conn(0x7fd22013b000 legacy=0x7fd22013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- 172.21.15.45:0/2076945505 shutdown_connections 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- 172.21.15.45:0/2076945505 >> 172.21.15.45:0/2076945505 conn(0x7fd2200a2bd0 msgr2=0x7fd2200a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- 172.21.15.45:0/2076945505 shutdown_connections 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- 172.21.15.45:0/2076945505 wait complete. 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: build_initial_monmap 2024-09-16T03:10:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: monmap: 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 librados: starting msgr at 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 librados: starting objecter 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 Processor -- start 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- start start 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 librados: setting wanted keys 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 librados: calling monclient init 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: init 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): picked mon.noname-b con 0x7fd22009b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:49.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): picked mon.noname-a con 0x7fd22009b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): picked mon.noname-c con 0x7fd22013b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd220094ab0 con 0x7fd22009b8e0 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22009d2c0 con 0x7fd22009b250 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22009ac20 con 0x7fd22013b000 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd227560640 10 monclient(hunting): authenticate will time out at 1340.599813s 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd224afb640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd22009b8e0 0x7fd2201d1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33600/0 (socket says 172.21.15.45:33600) 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.945+0000 7fd224afb640 1 -- 172.21.15.45:0/4194935501 learned_addr learned my addr 172.21.15.45:0/4194935501 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 client.?.objecter ms_handle_connect 0x7fd22009b8e0 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 client.?.objecter ms_handle_connect 0x7fd22013b000 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1842120677 0 0) 0x7fd22009ac20 con 0x7fd22013b000 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): my global_id is 114128 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd204004ec0 con 0x7fd22013b000 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832530180 0 0) 0x7fd220094ab0 con 0x7fd22009b8e0 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): my global_id is 104337 2024-09-16T03:10:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd22009ac20 con 0x7fd22009b8e0 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 client.?.objecter ms_handle_connect 0x7fd22009b250 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3223415230 0 0) 0x7fd204004ec0 con 0x7fd22013b000 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd220094ab0 con 0x7fd22013b000 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd2180018e0 con 0x7fd22013b000 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): dump: 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1208584842 0 0) 0x7fd22009ac20 con 0x7fd22009b8e0 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd204004ec0 con 0x7fd22009b8e0 2024-09-16T03:10:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 970020841 0 0) 0x7fd22009d2c0 con 0x7fd22009b250 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): my global_id is 22834 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd22009ac20 con 0x7fd22009b250 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd210001980 con 0x7fd22009b8e0 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient(hunting): dump: 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 326073268 0 0) 0x7fd220094ab0 con 0x7fd22013b000 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 >> v1:172.21.15.46:6789/0 conn(0x7fd22009b250 legacy=0x7fd2201ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 >> v1:172.21.15.45:6789/0 conn(0x7fd22009b8e0 legacy=0x7fd2201d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:10:49.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 monclient: found mon.c 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd220092750 con 0x7fd22013b000 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient: _finish_auth 0 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 5 monclient: authenticate success, global_id 114128 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 10 monclient: _renew_subs 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2200947d0 con 0x7fd22013b000 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 10 client.114128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 10 monclient: _renew_subs 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd220091dd0 con 0x7fd22013b000 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd218003c30 con 0x7fd22013b000 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 1 librados: init done 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd218003f80 con 0x7fd22013b000 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 monclient: dump: 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd2177fe640 4 set_mon_vals no callback set 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fd218001a80 con 0x7fd22013b000 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd22006c3e0 con 0x7fd22013b000 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(276..276 src has 1..276) ==== 6197+0+0 (unknown 943674340 0 0) 0x7fd21804aa30 con 0x7fd22013b000 2024-09-16T03:10:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 client.114128.objecter ms_dispatch 0x7fd220000ba0 osd_map(276..276 src has 1..276) 2024-09-16T03:10:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 3 client.114128.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T03:10:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 3 client.114128.objecter handle_osd_map decoding full epoch 276 2024-09-16T03:10:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 20 client.114128.objecter dump_active .. 0 homeless 2024-09-16T03:10:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd2252fc640 10 monclient: get_auth_request con 0x7fd20403e2b0 auth_method 0 2024-09-16T03:10:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.949+0000 7fd215ffb640 10 client.114128.objecter ms_handle_connect 0x7fd20403e2b0 2024-09-16T03:10:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.953+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd218015390 con 0x7fd22013b000 2024-09-16T03:10:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.953+0000 7fd215ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:49.953+0000 7fd215ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:50.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.193+0000 7fd227560640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:10:50.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.193+0000 7fd227560640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:10:50.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.193+0000 7fd227560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:50.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.193+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd220076990 con 0x7fd22013b000 2024-09-16T03:10:50.204 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.197+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.204 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.197+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=13,logm=55} crc {auth=1060605320,config=1097430493,health=3492118695,kv=2920680373,logm=755392081}) 2024-09-16T03:10:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.197+0000 7fd215ffb640 1 -- 172.21.15.45:0/4194935501 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fd21801ec40 con 0x7fd22013b000 2024-09-16T03:10:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.197+0000 7fd215ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:10:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.197+0000 7fd215ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:50.205 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.197+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=12,kv=13,logm=55} crc {auth=1060605320,config=1097430493,health=3492118695,kv=2264265189,logm=2262917034}) 2024-09-16T03:10:50.205 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.197+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.205 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.197+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=13,logm=55} crc {auth=1060605320,config=1097430493,health=3492118695,kv=2920680373,logm=755392081}) 2024-09-16T03:10:50.205 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.197+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=1060605320,config=1097430493,health=3077712967,kv=2038794863,logm=223763577}) 2024-09-16T03:10:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 10 librados: watch_flush enter 2024-09-16T03:10:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 10 librados: watch_flush exit 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 20 client.114128.objecter shutdown clearing up homeless session... 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 10 client.114128.objecter successfully canceled tick 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 >> v1:172.21.15.46:6812/149360280 conn(0x7fd20403e2b0 legacy=0x7fd204040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 10 monclient: shutdown 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 >> v1:172.21.15.45:6790/0 conn(0x7fd22013b000 legacy=0x7fd2201d5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 shutdown_connections 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 >> 172.21.15.45:0/4194935501 conn(0x7fd2200a2bd0 msgr2=0x7fd22009f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 shutdown_connections 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 1 -- 172.21.15.45:0/4194935501 wait complete. 2024-09-16T03:10:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:50.201+0000 7fd227560640 1 librados: shutdown 2024-09-16T03:10:50.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.201+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.201+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2535408404}) 2024-09-16T03:10:50.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.201+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3610114810}) 2024-09-16T03:10:50.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.201+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.201+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2535408404}) 2024-09-16T03:10:50.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.201+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688695165}) 2024-09-16T03:10:50.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730501250}) 2024-09-16T03:10:50.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273748018}) 2024-09-16T03:10:50.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730501250}) 2024-09-16T03:10:50.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316187592}) 2024-09-16T03:10:50.211 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824882062}) 2024-09-16T03:10:50.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2588094241}) 2024-09-16T03:10:50.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824882062}) 2024-09-16T03:10:50.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.205+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=884892209}) 2024-09-16T03:10:50.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863224634}) 2024-09-16T03:10:50.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2808598882}) 2024-09-16T03:10:50.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863224634}) 2024-09-16T03:10:50.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2637596965}) 2024-09-16T03:10:50.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030759266}) 2024-09-16T03:10:50.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3763469258}) 2024-09-16T03:10:50.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030759266}) 2024-09-16T03:10:50.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.209+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798182951}) 2024-09-16T03:10:50.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232099175}) 2024-09-16T03:10:50.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3767781725}) 2024-09-16T03:10:50.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232099175}) 2024-09-16T03:10:50.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1969340849}) 2024-09-16T03:10:50.220 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.220 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965899468}) 2024-09-16T03:10:50.221 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347037942}) 2024-09-16T03:10:50.221 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.221 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965899468}) 2024-09-16T03:10:50.221 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.213+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=875857749}) 2024-09-16T03:10:50.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.217+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.217+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236257151}) 2024-09-16T03:10:50.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.217+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2810620531}) 2024-09-16T03:10:50.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.217+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.217+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236257151}) 2024-09-16T03:10:50.224 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:10:50.224 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.217+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1330773357}) 2024-09-16T03:10:50.225 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.225 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360466453}) 2024-09-16T03:10:50.225 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4216483700}) 2024-09-16T03:10:50.225 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.225 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360466453}) 2024-09-16T03:10:50.225 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1238177522}) 2024-09-16T03:10:50.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135789943}) 2024-09-16T03:10:50.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1796249262}) 2024-09-16T03:10:50.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135789943}) 2024-09-16T03:10:50.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.221+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288389561}) 2024-09-16T03:10:50.229 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.229 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=312859080}) 2024-09-16T03:10:50.229 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=648710464}) 2024-09-16T03:10:50.229 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.229 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=312859080}) 2024-09-16T03:10:50.229 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348198140}) 2024-09-16T03:10:50.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209641208}) 2024-09-16T03:10:50.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780897232}) 2024-09-16T03:10:50.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209641208}) 2024-09-16T03:10:50.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.225+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1662861836}) 2024-09-16T03:10:50.233 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.233 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811981961}) 2024-09-16T03:10:50.233 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068300334}) 2024-09-16T03:10:50.233 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.234 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811981961}) 2024-09-16T03:10:50.234 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=197928029}) 2024-09-16T03:10:50.235 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.235 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3324070939}) 2024-09-16T03:10:50.235 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2364162733}) 2024-09-16T03:10:50.235 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.235 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3324070939}) 2024-09-16T03:10:50.235 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.229+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519349935}) 2024-09-16T03:10:50.237 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.237 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085562970}) 2024-09-16T03:10:50.237 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2179642090}) 2024-09-16T03:10:50.238 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.238 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085562970}) 2024-09-16T03:10:50.238 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990476582}) 2024-09-16T03:10:50.239 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.239 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16899119}) 2024-09-16T03:10:50.239 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2312138756}) 2024-09-16T03:10:50.239 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.239 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16899119}) 2024-09-16T03:10:50.240 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.233+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689798472}) 2024-09-16T03:10:50.241 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.241 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876479967}) 2024-09-16T03:10:50.241 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=945116980}) 2024-09-16T03:10:50.242 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.242 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876479967}) 2024-09-16T03:10:50.242 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527277644}) 2024-09-16T03:10:50.243 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.243 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1872331997}) 2024-09-16T03:10:50.243 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3016184590}) 2024-09-16T03:10:50.243 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.243 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1872331997}) 2024-09-16T03:10:50.243 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.237+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925866684}) 2024-09-16T03:10:50.245 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.245 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763070183}) 2024-09-16T03:10:50.245 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318436980}) 2024-09-16T03:10:50.246 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.246 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763070183}) 2024-09-16T03:10:50.246 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2803874584}) 2024-09-16T03:10:50.247 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.247 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191369502}) 2024-09-16T03:10:50.247 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3750200126}) 2024-09-16T03:10:50.247 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.247 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191369502}) 2024-09-16T03:10:50.247 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.241+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=952574747}) 2024-09-16T03:10:50.249 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.249 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386517346}) 2024-09-16T03:10:50.249 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3106506803}) 2024-09-16T03:10:50.249 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.250 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386517346}) 2024-09-16T03:10:50.250 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3796049391}) 2024-09-16T03:10:50.251 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.251 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332644668}) 2024-09-16T03:10:50.251 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583513289}) 2024-09-16T03:10:50.251 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.251 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332644668}) 2024-09-16T03:10:50.251 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.245+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=748094241}) 2024-09-16T03:10:50.253 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.253 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871664815}) 2024-09-16T03:10:50.253 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234464625}) 2024-09-16T03:10:50.253 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.253 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871664815}) 2024-09-16T03:10:50.253 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=573247969}) 2024-09-16T03:10:50.257 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.257 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878256848}) 2024-09-16T03:10:50.257 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=943217595}) 2024-09-16T03:10:50.257 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.257 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878256848}) 2024-09-16T03:10:50.257 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.249+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2553402897}) 2024-09-16T03:10:50.257 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.257 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849052727}) 2024-09-16T03:10:50.258 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2893251274}) 2024-09-16T03:10:50.258 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.258 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849052727}) 2024-09-16T03:10:50.258 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2956911004}) 2024-09-16T03:10:50.259 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.259 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184821346}) 2024-09-16T03:10:50.259 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=886906099}) 2024-09-16T03:10:50.259 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.259 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184821346}) 2024-09-16T03:10:50.260 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.253+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466933913}) 2024-09-16T03:10:50.261 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.261 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037579501}) 2024-09-16T03:10:50.261 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2276432417}) 2024-09-16T03:10:50.261 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.261 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037579501}) 2024-09-16T03:10:50.262 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2530579691}) 2024-09-16T03:10:50.263 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.263 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771442777}) 2024-09-16T03:10:50.263 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4233022200}) 2024-09-16T03:10:50.263 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.263 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771442777}) 2024-09-16T03:10:50.263 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.257+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226967967}) 2024-09-16T03:10:50.265 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.265 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621213704}) 2024-09-16T03:10:50.265 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2780693991}) 2024-09-16T03:10:50.265 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.265 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621213704}) 2024-09-16T03:10:50.265 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1027701399}) 2024-09-16T03:10:50.267 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.267 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094574140}) 2024-09-16T03:10:50.267 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=1050968829,mdsmap=2287870078,mgr=2286816098}) 2024-09-16T03:10:50.267 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.267 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094574140}) 2024-09-16T03:10:50.267 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.261+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=762915228}) 2024-09-16T03:10:50.270 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.265+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.270 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.265+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=2,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=18,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=18} crc {logm=714286497,mdsmap=4056336170,mgr=2950330609,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1453956271,mon_config_key=2387153988,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=3762605335,osd_snap=1739032456,osdmap=311981036}) 2024-09-16T03:10:50.270 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.265+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=20,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=27,osdmap=15} crc {logm=4057440968,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=56762826,mon_config_key=232079802,monmap=3372108098,nvmeofgw=3140514009,osd_metadata=1913529591,osd_pg_creating=3762605335,osd_snap=324269350,osdmap=2131611758}) 2024-09-16T03:10:50.270 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.265+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.271 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.265+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=2,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=18,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=18} crc {logm=714286497,mdsmap=4056336170,mgr=2950330609,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1453956271,mon_config_key=2387153988,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=3762605335,osd_snap=1739032456,osdmap=311981036}) 2024-09-16T03:10:50.271 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.265+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=2,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=14} crc {logm=4057440968,mdsmap=4056336170,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1453956271,mon_config_key=4020133541,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=3762605335,osd_snap=1923939111,osdmap=3487363866}) 2024-09-16T03:10:50.273 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.273 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=596311093}) 2024-09-16T03:10:50.273 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3192649633}) 2024-09-16T03:10:50.273 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.273 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=596311093}) 2024-09-16T03:10:50.273 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=953002027}) 2024-09-16T03:10:50.275 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.275 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1977182714}) 2024-09-16T03:10:50.275 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=725258673}) 2024-09-16T03:10:50.275 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.276 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1977182714}) 2024-09-16T03:10:50.276 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.269+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=309515471}) 2024-09-16T03:10:50.278 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.273+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.279 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.273+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=144658986}) 2024-09-16T03:10:50.279 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.273+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=653452782}) 2024-09-16T03:10:50.279 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.273+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.279 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.273+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=144658986}) 2024-09-16T03:10:50.279 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.273+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=558830936}) 2024-09-16T03:10:50.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.293+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.293+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=90} crc {osdmap=4049215849}) 2024-09-16T03:10:50.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.293+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=93} crc {osdmap=2936138111}) 2024-09-16T03:10:50.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.293+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:10:50.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.293+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=90} crc {osdmap=4049215849}) 2024-09-16T03:10:50.301 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:10:50.297+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=91} crc {osdmap=105385158}) 2024-09-16T03:10:51.225 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:10:51.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:10:51.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient: get_monmap_and_config 2024-09-16T03:10:51.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient: build_initial_monmap 2024-09-16T03:10:51.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient: monmap: 2024-09-16T03:10:51.358 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:51.359 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.359 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:51.357807+0000 2024-09-16T03:10:51.359 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:51.357807+0000 2024-09-16T03:10:51.359 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:51.359 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.359 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:51.360 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:51.360 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:51.360 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 1 Processor -- start 2024-09-16T03:10:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 1 -- start start 2024-09-16T03:10:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient: init 2024-09-16T03:10:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:10:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient(hunting): picked mon.noname-a con 0x7f8224135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient(hunting): picked mon.noname-c con 0x7f822409fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient(hunting): picked mon.noname-b con 0x7f822413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f822409f7e0 con 0x7f8224135950 2024-09-16T03:10:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f822409b160 con 0x7f822409fc40 2024-09-16T03:10:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f822409b440 con 0x7f822413c010 2024-09-16T03:10:51.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f82288fc640 10 monclient(hunting): authenticate will time out at 1342.007215s 2024-09-16T03:10:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822259c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8224135950 0x7f8224135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58324/0 (socket says 172.21.15.45:58324) 2024-09-16T03:10:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822259c640 1 -- 172.21.15.45:0/3653633916 learned_addr learned my addr 172.21.15.45:0/3653633916 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4009083126 0 0) 0x7f822409b440 con 0x7f822413c010 2024-09-16T03:10:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 10 monclient(hunting): my global_id is 114131 2024-09-16T03:10:51.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f820c003530 con 0x7f822413c010 2024-09-16T03:10:51.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 254326350 0 0) 0x7f822409f7e0 con 0x7f8224135950 2024-09-16T03:10:51.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 10 monclient(hunting): my global_id is 104340 2024-09-16T03:10:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.353+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f822409b440 con 0x7f8224135950 2024-09-16T03:10:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3702278392 0 0) 0x7f822409b160 con 0x7f822409fc40 2024-09-16T03:10:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient(hunting): my global_id is 22837 2024-09-16T03:10:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f822409f7e0 con 0x7f822409fc40 2024-09-16T03:10:51.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3969555495 0 0) 0x7f820c003530 con 0x7f822413c010 2024-09-16T03:10:51.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 >> v1:172.21.15.46:6789/0 conn(0x7f822409fc40 legacy=0x7f822413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 >> v1:172.21.15.45:6789/0 conn(0x7f8224135950 legacy=0x7f8224135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: _finish_hunting 0 2024-09-16T03:10:51.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 monclient: found mon.noname-b 2024-09-16T03:10:51.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:51.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8224136440 con 0x7f822413c010 2024-09-16T03:10:51.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: _finish_auth 0 2024-09-16T03:10:51.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: _check_auth_tickets 2024-09-16T03:10:51.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:51.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f821c001a20 con 0x7f822413c010 2024-09-16T03:10:51.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:10:51.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: dump: 2024-09-16T03:10:51.382 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.382 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.382 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.383 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.383 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.383 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.383 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.383 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.383 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.384 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 5 monclient: authenticate success, global_id 114131 2024-09-16T03:10:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f821c001bc0 con 0x7f822413c010 2024-09-16T03:10:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 1 -- 172.21.15.45:0/3653633916 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f821c002010 con 0x7f822413c010 2024-09-16T03:10:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 4 set_mon_vals no callback set 2024-09-16T03:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822159a640 10 monclient: dump: 2024-09-16T03:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.387 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.387 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.387 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.387 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.387 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.387 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient: shutdown 2024-09-16T03:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- 172.21.15.45:0/3653633916 >> v1:172.21.15.45:6790/0 conn(0x7f822413c010 legacy=0x7f822413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- 172.21.15.45:0/3653633916 shutdown_connections 2024-09-16T03:10:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- 172.21.15.45:0/3653633916 >> 172.21.15.45:0/3653633916 conn(0x7f82240a2bd0 msgr2=0x7f82240a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- 172.21.15.45:0/3653633916 shutdown_connections 2024-09-16T03:10:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- 172.21.15.45:0/3653633916 wait complete. 2024-09-16T03:10:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient: build_initial_monmap 2024-09-16T03:10:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient: monmap: 2024-09-16T03:10:51.390 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:51.390 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:51.390 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:51.390 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 librados: starting msgr at 2024-09-16T03:10:51.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 librados: starting objecter 2024-09-16T03:10:51.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 Processor -- start 2024-09-16T03:10:51.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- start start 2024-09-16T03:10:51.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 librados: setting wanted keys 2024-09-16T03:10:51.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 librados: calling monclient init 2024-09-16T03:10:51.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient: init 2024-09-16T03:10:51.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:51.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:10:51.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient(hunting): picked mon.noname-a con 0x7f822409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:51.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient(hunting): picked mon.noname-b con 0x7f8224135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:51.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient(hunting): picked mon.noname-c con 0x7f822413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:51.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8224071af0 con 0x7f822409fc40 2024-09-16T03:10:51.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f822406f1d0 con 0x7f8224135950 2024-09-16T03:10:51.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82240702f0 con 0x7f822413c010 2024-09-16T03:10:51.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:51.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient(hunting): authenticate will time out at 1342.010910s 2024-09-16T03:10:51.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822259c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f822409fc40 0x7f822406e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58334/0 (socket says 172.21.15.45:58334) 2024-09-16T03:10:51.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f822259c640 1 -- 172.21.15.45:0/2035387440 learned_addr learned my addr 172.21.15.45:0/2035387440 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:51.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 client.?.objecter ms_handle_connect 0x7f822409fc40 2024-09-16T03:10:51.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:51.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 client.?.objecter ms_handle_connect 0x7f822413c010 2024-09-16T03:10:51.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2594247153 0 0) 0x7f8224071af0 con 0x7f822409fc40 2024-09-16T03:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): my global_id is 104343 2024-09-16T03:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8204004ec0 con 0x7f822409fc40 2024-09-16T03:10:51.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 94935486 0 0) 0x7f82240702f0 con 0x7f822413c010 2024-09-16T03:10:51.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): my global_id is 114134 2024-09-16T03:10:51.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8224071af0 con 0x7f822413c010 2024-09-16T03:10:51.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 client.?.objecter ms_handle_connect 0x7f8224135950 2024-09-16T03:10:51.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:51.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1788592865 0 0) 0x7f8204004ec0 con 0x7f822409fc40 2024-09-16T03:10:51.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82240702f0 con 0x7f822409fc40 2024-09-16T03:10:51.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8218001a00 con 0x7f822409fc40 2024-09-16T03:10:51.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:51.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:51.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): dump: 2024-09-16T03:10:51.399 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.399 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 616543752 0 0) 0x7f8224071af0 con 0x7f822413c010 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8204004ec0 con 0x7f822413c010 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f821c001f10 con 0x7f822413c010 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): dump: 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4163677400 0 0) 0x7f822406f1d0 con 0x7f8224135950 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient(hunting): my global_id is 22840 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8224071af0 con 0x7f8224135950 2024-09-16T03:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4108317181 0 0) 0x7f8204004ec0 con 0x7f822413c010 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 >> v1:172.21.15.46:6789/0 conn(0x7f8224135950 legacy=0x7f822406eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 >> v1:172.21.15.45:6789/0 conn(0x7f822409fc40 legacy=0x7f822406e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 monclient: found mon.c 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822409d2c0 con 0x7f822413c010 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient: _finish_auth 0 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f8212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 5 monclient: authenticate success, global_id 114134 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient: _renew_subs 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.357+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f822406f790 con 0x7f822413c010 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f82288fc640 10 client.114134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f82288fc640 10 monclient: _renew_subs 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f82288fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f822413b2f0 con 0x7f822413c010 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f82288fc640 1 librados: init done 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f821c0020b0 con 0x7f822413c010 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f821c003cf0 con 0x7f822413c010 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 10 monclient: dump: 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8213fff640 4 set_mon_vals no callback set 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f82288fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f82288fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f82288fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8208005090 con 0x7f822413c010 2024-09-16T03:10:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f821c00fd60 con 0x7f822413c010 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(278..278 src has 1..278) ==== 5784+0+0 (unknown 2614093897 0 0) 0x7f821c04a9b0 con 0x7f822413c010 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 10 client.114134.objecter ms_dispatch 0x7f8224000ba0 osd_map(278..278 src has 1..278) 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 3 client.114134.objecter handle_osd_map got epochs [278,278] > 0 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 3 client.114134.objecter handle_osd_map decoding full epoch 278 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 20 client.114134.objecter dump_active .. 0 homeless 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f822259c640 10 monclient: get_auth_request con 0x7f820403e220 auth_method 0 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.361+0000 7f8212ffd640 10 client.114134.objecter ms_handle_connect 0x7f820403e220 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.365+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f821c015570 con 0x7f822413c010 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.365+0000 7f8212ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:51.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.365+0000 7f8212ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.605+0000 7f82288fc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:10:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.605+0000 7f82288fc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.605+0000 7f82288fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.605+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8208005380 con 0x7f822413c010 2024-09-16T03:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.605+0000 7f8212ffd640 1 -- 172.21.15.45:0/2035387440 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2297061825) 0x7f821c047f00 con 0x7f822413c010 2024-09-16T03:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.605+0000 7f8212ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.605+0000 7f8212ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:51.611 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:10:51.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 10 librados: watch_flush enter 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 10 librados: watch_flush exit 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 20 client.114134.objecter shutdown clearing up homeless session... 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 10 client.114134.objecter successfully canceled tick 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 >> v1:172.21.15.46:6812/149360280 conn(0x7f820403e220 legacy=0x7f82040406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 10 monclient: shutdown 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 >> v1:172.21.15.45:6790/0 conn(0x7f822413c010 legacy=0x7f82241d2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 shutdown_connections 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 >> 172.21.15.45:0/2035387440 conn(0x7f82240a2bd0 msgr2=0x7f822413ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 shutdown_connections 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 1 -- 172.21.15.45:0/2035387440 wait complete. 2024-09-16T03:10:51.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.609+0000 7f82288fc640 1 librados: shutdown 2024-09-16T03:10:51.630 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:10:51.631 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:10:51.631 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:10:51.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:10:51.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient: get_monmap_and_config 2024-09-16T03:10:51.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient: build_initial_monmap 2024-09-16T03:10:51.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient: monmap: 2024-09-16T03:10:51.753 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:51.753 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.753 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:51.752883+0000 2024-09-16T03:10:51.754 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:51.752883+0000 2024-09-16T03:10:51.754 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:51.754 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.754 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:51.754 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:51.754 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:51.754 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 1 Processor -- start 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 1 -- start start 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient: init 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc4135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc409f7e0 con 0x7fdbc409fc40 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc409b160 con 0x7fdbc4135950 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc409b440 con 0x7fdbc413c010 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:51.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 10 monclient(hunting): authenticate will time out at 1342.402989s 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc2ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdbc409fc40 0x7fdbc413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58338/0 (socket says 172.21.15.45:58338) 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc3fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdbc413c010 0x7fdbc413e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59774/0 (socket says 172.21.15.45:59774) 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc2ffd640 1 -- 172.21.15.45:0/3135431039 learned_addr learned my addr 172.21.15.45:0/3135431039 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029301053 0 0) 0x7fdbc409b440 con 0x7fdbc413c010 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient(hunting): my global_id is 114137 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbac003530 con 0x7fdbc413c010 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404709738 0 0) 0x7fdbc409f7e0 con 0x7fdbc409fc40 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient(hunting): my global_id is 104346 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc409b440 con 0x7fdbc409fc40 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1384731035 0 0) 0x7fdbac003530 con 0x7fdbc413c010 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 >> v1:172.21.15.46:6789/0 conn(0x7fdbc4135950 legacy=0x7fdbc4135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 >> v1:172.21.15.45:6789/0 conn(0x7fdbc409fc40 legacy=0x7fdbc413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient: _finish_hunting 0 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 monclient: found mon.noname-b 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc4136440 con 0x7fdbc413c010 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient: _finish_auth 0 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:10:51.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdbbc001a20 con 0x7fdbc413c010 2024-09-16T03:10:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:10:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc27fc640 10 monclient: dump: 2024-09-16T03:10:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.760 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 5 monclient: authenticate success, global_id 114137 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.749+0000 7fdbc9ced640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbbc001bc0 con 0x7fdbc413c010 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc27fc640 1 -- 172.21.15.45:0/3135431039 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdbbc002050 con 0x7fdbc413c010 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc27fc640 10 monclient: dump: 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 4 set_mon_vals no callback set 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: shutdown 2024-09-16T03:10:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:51.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- 172.21.15.45:0/3135431039 >> v1:172.21.15.45:6790/0 conn(0x7fdbc413c010 legacy=0x7fdbc413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- 172.21.15.45:0/3135431039 shutdown_connections 2024-09-16T03:10:51.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- 172.21.15.45:0/3135431039 >> 172.21.15.45:0/3135431039 conn(0x7fdbc40a2bd0 msgr2=0x7fdbc40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:51.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- 172.21.15.45:0/3135431039 shutdown_connections 2024-09-16T03:10:51.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- 172.21.15.45:0/3135431039 wait complete. 2024-09-16T03:10:51.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: build_initial_monmap 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: monmap: 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 librados: starting msgr at 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 librados: starting objecter 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 Processor -- start 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- start start 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 librados: setting wanted keys 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 librados: calling monclient init 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: init 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc4135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:51.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc413c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc40735e0 con 0x7fdbc413c010 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc409ed20 con 0x7fdbc4135950 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc409e480 con 0x7fdbc409fc40 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient(hunting): authenticate will time out at 1342.406214s 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc37fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdbc409fc40 0x7fdbc40727c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59784/0 (socket says 172.21.15.45:59784) 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc37fe640 1 -- 172.21.15.45:0/4054126083 learned_addr learned my addr 172.21.15.45:0/4054126083 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 client.?.objecter ms_handle_connect 0x7fdbc409fc40 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 client.?.objecter ms_handle_connect 0x7fdbc413c010 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715003563 0 0) 0x7fdbc409e480 con 0x7fdbc409fc40 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): my global_id is 114140 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb0004ec0 con 0x7fdbc409fc40 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383340307 0 0) 0x7fdbc40735e0 con 0x7fdbc413c010 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): my global_id is 104349 2024-09-16T03:10:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc409e480 con 0x7fdbc413c010 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 client.?.objecter ms_handle_connect 0x7fdbc4135950 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1012909881 0 0) 0x7fdbb0004ec0 con 0x7fdbc409fc40 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbc40735e0 con 0x7fdbc409fc40 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdbb4001a40 con 0x7fdbc409fc40 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): dump: 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.774 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1414914832 0 0) 0x7fdbc409e480 con 0x7fdbc413c010 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbb0004ec0 con 0x7fdbc413c010 2024-09-16T03:10:51.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdbbc001f10 con 0x7fdbc413c010 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): dump: 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.776 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3268335959 0 0) 0x7fdbc409ed20 con 0x7fdbc4135950 2024-09-16T03:10:51.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient(hunting): my global_id is 22846 2024-09-16T03:10:51.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc409e480 con 0x7fdbc4135950 2024-09-16T03:10:51.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1090459255 0 0) 0x7fdbc40735e0 con 0x7fdbc409fc40 2024-09-16T03:10:51.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 >> v1:172.21.15.46:6789/0 conn(0x7fdbc4135950 legacy=0x7fdbc4072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 >> v1:172.21.15.45:6789/0 conn(0x7fdbc413c010 legacy=0x7fdbc41d4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient: _finish_hunting 0 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 monclient: found mon.c 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc413b6b0 con 0x7fdbc409fc40 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient: _finish_auth 0 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient: _check_auth_tickets 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 5 monclient: authenticate success, global_id 114140 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: _renew_subs 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc409f140 con 0x7fdbc409fc40 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 client.114140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: _renew_subs 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbc41d64d0 con 0x7fdbc409fc40 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 librados: init done 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbb4001d40 con 0x7fdbc409fc40 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdbb4003a70 con 0x7fdbc409fc40 2024-09-16T03:10:51.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc1ffb640 4 set_mon_vals no callback set 2024-09-16T03:10:51.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:51.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbabfff640 10 monclient: dump: 2024-09-16T03:10:51.781 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.781 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.781 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.781 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.781 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.753+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdba40065d0 con 0x7fdba4002870 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 client.114140.objecter ms_handle_connect 0x7fdba4002870 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 client.114140.objecter resend_mon_ops 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fdbb4003c10 con 0x7fdbc409fc40 2024-09-16T03:10:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(278..278 src has 1..278) ==== 5784+0+0 (unknown 2614093897 0 0) 0x7fdbb404ba00 con 0x7fdbc409fc40 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 client.114140.objecter ms_dispatch 0x7fdbc4000ba0 osd_map(278..278 src has 1..278) 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 3 client.114140.objecter handle_osd_map got epochs [278,278] > 0 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 3 client.114140.objecter handle_osd_map decoding full epoch 278 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 20 client.114140.objecter dump_active .. 0 homeless 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 451228352 0 0) 0x7fdba40065d0 con 0x7fdba4002870 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 monclient(hunting): my global_id is 104352 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc40735e0 con 0x7fdba4002870 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2863888884 0 0) 0x7fdbc40735e0 con 0x7fdba4002870 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdba40065d0 con 0x7fdba4002870 2024-09-16T03:10:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbc3fff640 10 monclient: get_auth_request con 0x7fdbb003e230 auth_method 0 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdbb8000f70 con 0x7fdba4002870 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 monclient: dump: 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3417032938 0 0) 0x7fdba40065d0 con 0x7fdba4002870 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdba400a170 con 0x7fdba4002870 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 client.114140.objecter ms_handle_connect 0x7fdbb003e230 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdbb8001ce0 con 0x7fdba4002870 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 >> v1:172.21.15.45:6789/0 conn(0x7fdba4002870 legacy=0x7fdba4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.757+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 >> v1:172.21.15.45:6789/0 conn(0x7fdba4002870 legacy=0x7fdba4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.761+0000 7fdbc9ced640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.761+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdba4009e80 con 0x7fdba4009140 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.761+0000 7fdbabfff640 10 client.114140.objecter ms_handle_connect 0x7fdba4009140 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 client.114140.objecter resend_mon_ops 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 758867416 0 0) 0x7fdba4009e80 con 0x7fdba4009140 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 monclient(hunting): my global_id is 104355 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba40065d0 con 0x7fdba4009140 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3362173945 0 0) 0x7fdba40065d0 con 0x7fdba4009140 2024-09-16T03:10:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdba4009e80 con 0x7fdba4009140 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdbb8004990 con 0x7fdba4009140 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 monclient: dump: 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.787 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1400605832 0 0) 0x7fdba4009e80 con 0x7fdba4009140 2024-09-16T03:10:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdba400c420 con 0x7fdba4009140 2024-09-16T03:10:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4137 (unknown 0 0 2082752510) 0x7fdbb8004920 con 0x7fdba4009140 2024-09-16T03:10:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:10:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 >> v1:172.21.15.45:6789/0 conn(0x7fdba4009140 legacy=0x7fdba40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbabfff640 1 -- 172.21.15.45:0/4054126083 >> v1:172.21.15.45:6789/0 conn(0x7fdba4009140 legacy=0x7fdba40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:51.788 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 26, 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 110866, 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:51.789 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:51.790 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:51.791 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:10:51.792 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:51.793 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:51.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.795 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:10:51.796 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.799 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:51.800 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:51.800 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:10:51.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.800 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.800 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:10:51.800 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:10:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 10 librados: watch_flush enter 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 10 librados: watch_flush exit 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 20 client.114140.objecter shutdown clearing up homeless session... 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 10 client.114140.objecter successfully canceled tick 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 >> v1:172.21.15.46:6812/149360280 conn(0x7fdbb003e230 legacy=0x7fdbb00406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 10 monclient: shutdown 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 >> v1:172.21.15.45:6790/0 conn(0x7fdbc409fc40 legacy=0x7fdbc40727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc3fff640 1 -- 172.21.15.45:0/4054126083 reap_dead start 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc3fff640 1 -- 172.21.15.45:0/4054126083 reap_dead start 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 shutdown_connections 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 >> 172.21.15.45:0/4054126083 conn(0x7fdbc40a2bd0 msgr2=0x7fdbc413ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 shutdown_connections 2024-09-16T03:10:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 1 -- 172.21.15.45:0/4054126083 wait complete. 2024-09-16T03:10:51.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.765+0000 7fdbc9ced640 1 librados: shutdown 2024-09-16T03:10:51.803 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:10:51.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:10:51.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:10:51.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.909+0000 7f1cf8fdb640 10 monclient: get_monmap_and_config 2024-09-16T03:10:51.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.909+0000 7f1cf8fdb640 10 monclient: build_initial_monmap 2024-09-16T03:10:51.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.909+0000 7f1cf8fdb640 10 monclient: monmap: 2024-09-16T03:10:51.915 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:51.915 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.915 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:51.915167+0000 2024-09-16T03:10:51.916 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:51.915167+0000 2024-09-16T03:10:51.916 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:51.916 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.916 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:51.916 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:51.916 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:51.916 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 Processor -- start 2024-09-16T03:10:51.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- start start 2024-09-16T03:10:51.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient: init 2024-09-16T03:10:51.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:51.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:10:51.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient(hunting): picked mon.noname-b con 0x7f1cf409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient(hunting): picked mon.noname-a con 0x7f1cf409b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient(hunting): picked mon.noname-c con 0x7f1cf413b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf409cc20 con 0x7f1cf409b250 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf409d820 con 0x7f1cf413b000 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf409b6a0 con 0x7f1cf409b8e0 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient(hunting): authenticate will time out at 1342.564948s 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf1d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1cf409b250 0x7f1cf41398e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58388/0 (socket says 172.21.15.45:58388) 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf1d9b640 1 -- 172.21.15.45:0/480764179 learned_addr learned my addr 172.21.15.45:0/480764179 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf259c640 1 --1- 172.21.15.45:0/480764179 >> v1:172.21.15.45:6790/0 conn(0x7f1cf409b8e0 0x7f1cf409bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59800/0 (socket says 172.21.15.45:59800) 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3906863729 0 0) 0x7f1cf409b6a0 con 0x7f1cf409b8e0 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient(hunting): my global_id is 114143 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cdc005740 con 0x7f1cf409b8e0 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2510568016 0 0) 0x7f1cf409cc20 con 0x7f1cf409b250 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient(hunting): my global_id is 104358 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cf409b6a0 con 0x7f1cf409b250 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3874988944 0 0) 0x7f1cdc005740 con 0x7f1cf409b8e0 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 >> v1:172.21.15.46:6789/0 conn(0x7f1cf413b000 legacy=0x7f1cf413d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 >> v1:172.21.15.45:6789/0 conn(0x7f1cf409b250 legacy=0x7f1cf41398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: _finish_hunting 0 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 monclient: found mon.noname-b 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf40a21f0 con 0x7f1cf409b8e0 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: _finish_auth 0 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: _check_auth_tickets 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1ce4001a20 con 0x7f1cf409b8e0 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: dump: 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.922 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 5 monclient: authenticate success, global_id 114143 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ce4001bc0 con 0x7f1cf409b8e0 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:51.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 1 -- 172.21.15.45:0/480764179 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1ce4001fd0 con 0x7f1cf409b8e0 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf159a640 10 monclient: dump: 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.924 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 4 set_mon_vals no callback set 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient: shutdown 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/480764179 >> v1:172.21.15.45:6790/0 conn(0x7f1cf409b8e0 legacy=0x7f1cf409bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/480764179 shutdown_connections 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/480764179 >> 172.21.15.45:0/480764179 conn(0x7f1cf40a2bd0 msgr2=0x7f1cf40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/480764179 shutdown_connections 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/480764179 wait complete. 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient: build_initial_monmap 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient: monmap: 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 librados: starting msgr at 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 librados: starting objecter 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 Processor -- start 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 -- start start 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 librados: setting wanted keys 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 1 librados: calling monclient init 2024-09-16T03:10:51.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.913+0000 7f1cf8fdb640 10 monclient: init 2024-09-16T03:10:51.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:51.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient(hunting): picked mon.noname-a con 0x7f1cf409b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient(hunting): picked mon.noname-c con 0x7f1cf409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient(hunting): picked mon.noname-b con 0x7f1cf413b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf4071da0 con 0x7f1cf409b250 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf40a5290 con 0x7f1cf413b000 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf4074100 con 0x7f1cf409b8e0 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient(hunting): authenticate will time out at 1342.568129s 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf259c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1cf409b250 0x7f1cf4070410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58398/0 (socket says 172.21.15.45:58398) 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf259c640 1 -- 172.21.15.45:0/354092678 learned_addr learned my addr 172.21.15.45:0/354092678 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf1d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1cf409b8e0 0x7f1cf4070b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59808/0 (socket says 172.21.15.45:59808) 2024-09-16T03:10:51.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f1cf409b250 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f1cf409b8e0 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2633074039 0 0) 0x7f1cf4071da0 con 0x7f1cf409b250 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): my global_id is 104361 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cd4004ec0 con 0x7f1cf409b250 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2475070184 0 0) 0x7f1cf4074100 con 0x7f1cf409b8e0 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): my global_id is 114146 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cf4071da0 con 0x7f1cf409b8e0 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f1cf413b000 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2678135973 0 0) 0x7f1cd4004ec0 con 0x7f1cf409b250 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cf4074100 con 0x7f1cf409b250 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1ce4001f50 con 0x7f1cf409b250 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): dump: 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.935 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 844708008 0 0) 0x7f1cf4071da0 con 0x7f1cf409b8e0 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cd4004ec0 con 0x7f1cf409b8e0 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1ce8001140 con 0x7f1cf409b8e0 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): dump: 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736118584 0 0) 0x7f1cf409d820 con 0x7f1cf413b000 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): my global_id is 22852 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cf4071da0 con 0x7f1cf413b000 2024-09-16T03:10:51.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4243890657 0 0) 0x7f1cf4074100 con 0x7f1cf409b250 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 >> v1:172.21.15.45:6790/0 conn(0x7f1cf409b8e0 legacy=0x7f1cf4070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 >> v1:172.21.15.46:6789/0 conn(0x7f1cf413b000 legacy=0x7f1cf41d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 monclient: found mon.a 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf4071230 con 0x7f1cf409b250 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 5 monclient: authenticate success, global_id 104361 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient: _renew_subs 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:51.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf413abd0 con 0x7f1cf409b250 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 client.104361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient: _renew_subs 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1cf4072900 con 0x7f1cf409b250 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 1 librados: init done 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ce4004ec0 con 0x7f1cf409b250 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1ce4005210 con 0x7f1cf409b250 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf0d99640 4 set_mon_vals no callback set 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: dump: 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.939 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1cd80065d0 con 0x7f1cd8002870 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.104361.objecter ms_handle_connect 0x7f1cd8002870 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.104361.objecter resend_mon_ops 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235070889 0 0) 0x7f1cd80065d0 con 0x7f1cd8002870 2024-09-16T03:10:51.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient(hunting): my global_id is 114149 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cf4074100 con 0x7f1cd8002870 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f1ce4011280 con 0x7f1cf409b250 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(278..278 src has 1..278) ==== 5784+0+0 (unknown 2614093897 0 0) 0x7f1ce404bbe0 con 0x7f1cf409b250 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.104361.objecter ms_dispatch 0x7f1cf4000ba0 osd_map(278..278 src has 1..278) 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 3 client.104361.objecter handle_osd_map got epochs [278,278] > 0 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 3 client.104361.objecter handle_osd_map decoding full epoch 278 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 20 client.104361.objecter dump_active .. 0 homeless 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3266780932 0 0) 0x7f1cf4074100 con 0x7f1cd8002870 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cd80065d0 con 0x7f1cd8002870 2024-09-16T03:10:51.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1ce80047f0 con 0x7f1cd8002870 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: dump: 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1cf2d9d640 10 monclient: get_auth_request con 0x7f1cd403e300 auth_method 0 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1998115822 0 0) 0x7f1cd80065d0 con 0x7f1cd8002870 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1cd800a170 con 0x7f1cd8002870 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 client.104361.objecter ms_handle_connect 0x7f1cd403e300 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1ce8004850 con 0x7f1cd8002870 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 >> v1:172.21.15.45:6790/0 conn(0x7f1cd8002870 legacy=0x7f1cd8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.917+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 >> v1:172.21.15.45:6790/0 conn(0x7f1cd8002870 legacy=0x7f1cd8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1cd8009e80 con 0x7f1cd8009140 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 client.104361.objecter ms_handle_connect 0x7f1cd8009140 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 client.104361.objecter resend_mon_ops 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1663225938 0 0) 0x7f1cd8009e80 con 0x7f1cd8009140 2024-09-16T03:10:51.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 monclient(hunting): my global_id is 114152 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cd80065d0 con 0x7f1cd8009140 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 730509189 0 0) 0x7f1cd80065d0 con 0x7f1cd8009140 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cd8009e80 con 0x7f1cd8009140 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1ce8004fc0 con 0x7f1cd8009140 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 monclient: dump: 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:51.944 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2666006917 0 0) 0x7f1cd8009e80 con 0x7f1cd8009140 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1cd800c400 con 0x7f1cd8009140 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4133 (unknown 0 0 3851857925) 0x7f1ce8007280 con 0x7f1cd8009140 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:51.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 >> v1:172.21.15.45:6790/0 conn(0x7f1cd8009140 legacy=0x7f1cd80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/354092678 >> v1:172.21.15.45:6790/0 conn(0x7f1cd8009140 legacy=0x7f1cd80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:51.946 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:10:51.946 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:10:51.946 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:10:51.946 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:10:51.946 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 26, 2024-09-16T03:10:51.946 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 2987, 2024-09-16T03:10:51.946 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:10:51.946 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:51.947 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:51.948 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:10:51.949 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:51.950 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:10:51.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:10:51.952 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:10:51.953 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:51.954 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:51.956 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:10:51.957 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:10:51.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 10 librados: watch_flush enter 2024-09-16T03:10:51.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 10 librados: watch_flush exit 2024-09-16T03:10:51.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 20 client.104361.objecter shutdown clearing up homeless session... 2024-09-16T03:10:51.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 10 client.104361.objecter successfully canceled tick 2024-09-16T03:10:51.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 >> v1:172.21.15.46:6812/149360280 conn(0x7f1cd403e300 legacy=0x7f1cd40407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 10 monclient: shutdown 2024-09-16T03:10:51.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:51.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 >> v1:172.21.15.45:6789/0 conn(0x7f1cf409b250 legacy=0x7f1cf4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:51.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf2d9d640 1 -- 172.21.15.45:0/354092678 reap_dead start 2024-09-16T03:10:51.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf2d9d640 1 -- 172.21.15.45:0/354092678 reap_dead start 2024-09-16T03:10:51.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 shutdown_connections 2024-09-16T03:10:51.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 >> 172.21.15.45:0/354092678 conn(0x7f1cf40a2bd0 msgr2=0x7f1cf40a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:51.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 shutdown_connections 2024-09-16T03:10:51.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 1 -- 172.21.15.45:0/354092678 wait complete. 2024-09-16T03:10:51.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:51.925+0000 7f1cf8fdb640 1 librados: shutdown 2024-09-16T03:10:51.961 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:10:51.961 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:10:51.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient: get_monmap_and_config 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient: build_initial_monmap 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient: monmap: 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:52.076821+0000 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:52.076821+0000 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:52.077 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:52.078 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 1 Processor -- start 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 1 -- start start 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient: init 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient(hunting): picked mon.noname-c con 0x7f18a8135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient(hunting): picked mon.noname-b con 0x7f18a809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient(hunting): picked mon.noname-a con 0x7f18a813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a809f7e0 con 0x7f18a813c010 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a809b160 con 0x7f18a8135950 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a809b440 con 0x7f18a809fc40 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:52.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient(hunting): authenticate will time out at 1342.726223s 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a77fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f18a809fc40 0x7f18a813a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59844/0 (socket says 172.21.15.45:59844) 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ac906640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f18a813c010 0x7f18a813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58400/0 (socket says 172.21.15.45:58400) 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a77fe640 1 -- 172.21.15.45:0/1047738867 learned_addr learned my addr 172.21.15.45:0/1047738867 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2671438130 0 0) 0x7f18a809f7e0 con 0x7f18a813c010 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): my global_id is 104364 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1890003530 con 0x7f18a813c010 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1451206587 0 0) 0x7f18a809b440 con 0x7f18a809fc40 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): my global_id is 114155 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a809f7e0 con 0x7f18a809fc40 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1256237712 0 0) 0x7f18a809b160 con 0x7f18a8135950 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient(hunting): my global_id is 22855 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a809b440 con 0x7f18a8135950 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4199446804 0 0) 0x7f1890003530 con 0x7f18a813c010 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 >> v1:172.21.15.45:6790/0 conn(0x7f18a809fc40 legacy=0x7f18a813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 >> v1:172.21.15.46:6789/0 conn(0x7f18a8135950 legacy=0x7f18a8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 monclient: found mon.noname-a 2024-09-16T03:10:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a8136440 con 0x7f18a813c010 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f18a0001a20 con 0x7f18a813c010 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: dump: 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 5 monclient: authenticate success, global_id 104364 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18a0001c40 con 0x7f18a813c010 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 4 set_mon_vals no callback set 2024-09-16T03:10:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 1 -- 172.21.15.45:0/1047738867 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f18a0002090 con 0x7f18a813c010 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18a6ffd640 10 monclient: dump: 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.086 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 10 monclient: shutdown 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.073+0000 7f18ae369640 1 -- 172.21.15.45:0/1047738867 >> v1:172.21.15.45:6789/0 conn(0x7f18a813c010 legacy=0x7f18a813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- 172.21.15.45:0/1047738867 shutdown_connections 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- 172.21.15.45:0/1047738867 >> 172.21.15.45:0/1047738867 conn(0x7f18a80a2bd0 msgr2=0x7f18a80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- 172.21.15.45:0/1047738867 shutdown_connections 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- 172.21.15.45:0/1047738867 wait complete. 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient: build_initial_monmap 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient: monmap: 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:52.088 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 librados: starting msgr at 2024-09-16T03:10:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 librados: starting objecter 2024-09-16T03:10:52.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 Processor -- start 2024-09-16T03:10:52.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- start start 2024-09-16T03:10:52.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 librados: setting wanted keys 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 librados: calling monclient init 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient: init 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient(hunting): picked mon.noname-a con 0x7f18a809fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient(hunting): picked mon.noname-c con 0x7f18a8135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient(hunting): picked mon.noname-b con 0x7f18a813c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a8072d10 con 0x7f18a809fc40 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a8072ff0 con 0x7f18a813c010 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a80732d0 con 0x7f18a8135950 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient(hunting): authenticate will time out at 1342.729737s 2024-09-16T03:10:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f18a809fc40 0x7f18a80994f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58414/0 (socket says 172.21.15.45:58414) 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a7fff640 1 -- 172.21.15.45:0/1664239925 learned_addr learned my addr 172.21.15.45:0/1664239925 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 client.?.objecter ms_handle_connect 0x7f18a8135950 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 client.?.objecter ms_handle_connect 0x7f18a809fc40 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1689338099 0 0) 0x7f18a80732d0 con 0x7f18a8135950 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): my global_id is 114158 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1894004ec0 con 0x7f18a8135950 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 768468002 0 0) 0x7f18a8072d10 con 0x7f18a809fc40 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): my global_id is 104367 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a80732d0 con 0x7f18a809fc40 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 client.?.objecter ms_handle_connect 0x7f18a813c010 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2818920089 0 0) 0x7f1894004ec0 con 0x7f18a8135950 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18a8072d10 con 0x7f18a8135950 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2253405375 0 0) 0x7f18a80732d0 con 0x7f18a809fc40 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1894004ec0 con 0x7f18a809fc40 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f189c0018c0 con 0x7f18a8135950 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:10:52.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): dump: 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1898001980 con 0x7f18a809fc40 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:52.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): dump: 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975122880 0 0) 0x7f18a8072ff0 con 0x7f18a813c010 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient(hunting): my global_id is 22858 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a80732d0 con 0x7f18a813c010 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1215714216 0 0) 0x7f18a8072d10 con 0x7f18a8135950 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 >> v1:172.21.15.46:6789/0 conn(0x7f18a813c010 legacy=0x7f18a81d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 >> v1:172.21.15.45:6789/0 conn(0x7f18a809fc40 legacy=0x7f18a80994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 monclient: found mon.c 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a8071e40 con 0x7f18a8135950 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient: _finish_auth 0 2024-09-16T03:10:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 5 monclient: authenticate success, global_id 114158 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient: _renew_subs 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18a80712d0 con 0x7f18a8135950 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 client.114158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient: _renew_subs 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f18a8099dd0 con 0x7f18a8135950 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 librados: init done 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f189c0047f0 con 0x7f18a8135950 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f189c004b40 con 0x7f18a8135950 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a67fc640 4 set_mon_vals no callback set 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 10 monclient: dump: 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.101 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f18880065d0 con 0x7f1888002870 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.077+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f189c010bb0 con 0x7f18a8135950 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(279..279 src has 1..279) ==== 5457+0+0 (unknown 1445761015 0 0) 0x7f189c04b770 con 0x7f18a8135950 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 client.114158.objecter ms_dispatch 0x7f18a8000ba0 osd_map(279..279 src has 1..279) 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 3 client.114158.objecter handle_osd_map got epochs [279,279] > 0 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 3 client.114158.objecter handle_osd_map decoding full epoch 279 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 20 client.114158.objecter dump_active .. 0 homeless 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 client.114158.objecter ms_handle_connect 0x7f1888002870 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 client.114158.objecter resend_mon_ops 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18ac906640 10 monclient: get_auth_request con 0x7f189403e240 auth_method 0 2024-09-16T03:10:52.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 284648586 0 0) 0x7f18880065d0 con 0x7f1888002870 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 monclient(hunting): my global_id is 22861 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a8072d10 con 0x7f1888002870 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1945306795 0 0) 0x7f18a8072d10 con 0x7f1888002870 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18880065d0 con 0x7f1888002870 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1898003a70 con 0x7f1888002870 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 monclient: dump: 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.103 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 client.114158.objecter ms_handle_connect 0x7f189403e240 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1000131702 0 0) 0x7f18880065d0 con 0x7f1888002870 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f188800a170 con 0x7f1888002870 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f18980057d0 con 0x7f1888002870 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:52.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:52.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 >> v1:172.21.15.46:6789/0 conn(0x7f1888002870 legacy=0x7f1888004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.081+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 >> v1:172.21.15.46:6789/0 conn(0x7f1888002870 legacy=0x7f1888004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:52.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f188800c500 con 0x7f18880090f0 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 client.114158.objecter ms_handle_connect 0x7f18880090f0 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 client.114158.objecter resend_mon_ops 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824385745 0 0) 0x7f188800c500 con 0x7f18880090f0 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 monclient(hunting): my global_id is 22864 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18880065d0 con 0x7f18880090f0 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3152138479 0 0) 0x7f18880065d0 con 0x7f18880090f0 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f188800c500 con 0x7f18880090f0 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1898004160 con 0x7f18880090f0 2024-09-16T03:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 monclient: dump: 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 591778017 0 0) 0x7f188800c500 con 0x7f18880090f0 2024-09-16T03:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1888009ca0 con 0x7f18880090f0 2024-09-16T03:10:52.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 3842780935) 0x7f18980041e0 con 0x7f18880090f0 2024-09-16T03:10:52.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:10:52.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:52.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 >> v1:172.21.15.46:6789/0 conn(0x7f18880090f0 legacy=0x7f1888009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18a4ff9640 1 -- 172.21.15.45:0/1664239925 >> v1:172.21.15.46:6789/0 conn(0x7f18880090f0 legacy=0x7f1888009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 26, 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 373899, 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:10:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 3, 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:10:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:10:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:10:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:10:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:10:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:10:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:10:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:10:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:10:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:10:52.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 10 librados: watch_flush enter 2024-09-16T03:10:52.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 10 librados: watch_flush exit 2024-09-16T03:10:52.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 20 client.114158.objecter shutdown clearing up homeless session... 2024-09-16T03:10:52.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 10 client.114158.objecter successfully canceled tick 2024-09-16T03:10:52.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 >> v1:172.21.15.46:6812/149360280 conn(0x7f189403e240 legacy=0x7f1894040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 10 monclient: shutdown 2024-09-16T03:10:52.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:52.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 >> v1:172.21.15.45:6790/0 conn(0x7f18a8135950 legacy=0x7f18a81cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ac906640 1 -- 172.21.15.45:0/1664239925 reap_dead start 2024-09-16T03:10:52.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ac906640 1 -- 172.21.15.45:0/1664239925 reap_dead start 2024-09-16T03:10:52.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 shutdown_connections 2024-09-16T03:10:52.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 >> 172.21.15.45:0/1664239925 conn(0x7f18a80a2bd0 msgr2=0x7f18a80a0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:52.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 shutdown_connections 2024-09-16T03:10:52.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 1 -- 172.21.15.45:0/1664239925 wait complete. 2024-09-16T03:10:52.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.089+0000 7f18ae369640 1 librados: shutdown 2024-09-16T03:10:52.123 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:10:52.124 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:10:52.124 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:10:52.124 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:10:52.124 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:10:52.124 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:10:52.124 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:10:52.136 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:10:52.136 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:10:52.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:10:52.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient: get_monmap_and_config 2024-09-16T03:10:52.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient: build_initial_monmap 2024-09-16T03:10:52.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient: monmap: 2024-09-16T03:10:52.298 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:52.298 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:52.298156+0000 2024-09-16T03:10:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:52.298156+0000 2024-09-16T03:10:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:52.299 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 1 Processor -- start 2024-09-16T03:10:52.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 1 -- start start 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient: init 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient(hunting): picked mon.noname-b con 0x7f6a84135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient(hunting): picked mon.noname-a con 0x7f6a8409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient(hunting): picked mon.noname-c con 0x7f6a8413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a8409f7e0 con 0x7f6a8409fc40 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a8409b160 con 0x7f6a8413c010 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a8409b440 con 0x7f6a84135950 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8c4b6640 10 monclient(hunting): authenticate will time out at 1342.947142s 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a8a252640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f6a84135950 legacy=0x7f6a84135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:10:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a89a51640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6a8409fc40 0x7f6a8413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58420/0 (socket says 172.21.15.45:58420) 2024-09-16T03:10:52.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a89a51640 1 -- 172.21.15.45:0/3585914447 learned_addr learned my addr 172.21.15.45:0/3585914447 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:52.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211149360 0 0) 0x7f6a8409f7e0 con 0x7f6a8409fc40 2024-09-16T03:10:52.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a89250640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a89250640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a89250640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a89250640 10 monclient(hunting): my global_id is 104370 2024-09-16T03:10:52.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.293+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a70003530 con 0x7f6a8409fc40 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412876491 0 0) 0x7f6a8409b160 con 0x7f6a8413c010 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient(hunting): my global_id is 22867 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a8409f7e0 con 0x7f6a8413c010 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3518421140 0 0) 0x7f6a70003530 con 0x7f6a8409fc40 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 >> v1:172.21.15.45:6790/0 conn(0x7f6a84135950 legacy=0x7f6a84135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 >> v1:172.21.15.46:6789/0 conn(0x7f6a8413c010 legacy=0x7f6a8413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: _finish_hunting 0 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 monclient: found mon.noname-a 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:52.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:10:52.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a84136440 con 0x7f6a8409fc40 2024-09-16T03:10:52.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: _finish_auth 0 2024-09-16T03:10:52.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: _check_auth_tickets 2024-09-16T03:10:52.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:52.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a800016a0 con 0x7f6a8409fc40 2024-09-16T03:10:52.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: dump: 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.313 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 5 monclient: authenticate success, global_id 104370 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a80001840 con 0x7f6a8409fc40 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 1 -- 172.21.15.45:0/3585914447 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a80001c90 con 0x7f6a8409fc40 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a89250640 10 monclient: dump: 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.314 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.315 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.315 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.315 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.315 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.315 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:52.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 4 set_mon_vals no callback set 2024-09-16T03:10:52.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: shutdown 2024-09-16T03:10:52.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:52.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/3585914447 >> v1:172.21.15.45:6789/0 conn(0x7f6a8409fc40 legacy=0x7f6a8413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/3585914447 shutdown_connections 2024-09-16T03:10:52.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/3585914447 >> 172.21.15.45:0/3585914447 conn(0x7f6a840a2bd0 msgr2=0x7f6a840a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:52.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/3585914447 shutdown_connections 2024-09-16T03:10:52.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/3585914447 wait complete. 2024-09-16T03:10:52.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: build_initial_monmap 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: monmap: 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 librados: starting msgr at 2024-09-16T03:10:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 librados: starting objecter 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 Processor -- start 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- start start 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 librados: setting wanted keys 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 librados: calling monclient init 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: init 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient(hunting): picked mon.noname-c con 0x7f6a8409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient(hunting): picked mon.noname-b con 0x7f6a84135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient(hunting): picked mon.noname-a con 0x7f6a8413c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a840735e0 con 0x7f6a8413c010 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a8409d2c0 con 0x7f6a84135950 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a84073bf0 con 0x7f6a8409fc40 2024-09-16T03:10:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8a252640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f6a8409fc40 legacy=0x7f6a840727c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient(hunting): authenticate will time out at 1342.950397s 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8aa53640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6a8413c010 0x7f6a841d4db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58436/0 (socket says 172.21.15.45:58436) 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8aa53640 1 -- 172.21.15.45:0/47235060 learned_addr learned my addr 172.21.15.45:0/47235060 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 client.?.objecter ms_handle_connect 0x7f6a8413c010 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 654151792 0 0) 0x7f6a840735e0 con 0x7f6a8413c010 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): my global_id is 104373 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a68004ec0 con 0x7f6a8413c010 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 client.?.objecter ms_handle_connect 0x7f6a84135950 2024-09-16T03:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3989741460 0 0) 0x7f6a68004ec0 con 0x7f6a8413c010 2024-09-16T03:10:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a840735e0 con 0x7f6a8413c010 2024-09-16T03:10:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a7c001d20 con 0x7f6a8413c010 2024-09-16T03:10:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): dump: 2024-09-16T03:10:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315690394 0 0) 0x7f6a8409d2c0 con 0x7f6a84135950 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): _negotiate 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient(hunting): my global_id is 22870 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a68004ec0 con 0x7f6a84135950 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3469649091 0 0) 0x7f6a840735e0 con 0x7f6a8413c010 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 >> v1:172.21.15.45:6790/0 conn(0x7f6a8409fc40 legacy=0x7f6a840727c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 >> v1:172.21.15.46:6789/0 conn(0x7f6a84135950 legacy=0x7f6a84072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient: _finish_hunting 0 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 monclient: found mon.a 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8409f010 con 0x7f6a8413c010 2024-09-16T03:10:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient: _finish_auth 0 2024-09-16T03:10:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient: _check_auth_tickets 2024-09-16T03:10:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 5 monclient: authenticate success, global_id 104373 2024-09-16T03:10:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: _renew_subs 2024-09-16T03:10:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:52.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8409d810 con 0x7f6a8413c010 2024-09-16T03:10:52.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 client.104373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:52.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: _renew_subs 2024-09-16T03:10:52.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:52.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a8413b2f0 con 0x7f6a8413c010 2024-09-16T03:10:52.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a8c4b6640 1 librados: init done 2024-09-16T03:10:52.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a7c002060 con 0x7f6a8413c010 2024-09-16T03:10:52.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:52.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a7c003df0 con 0x7f6a8413c010 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a88a4f640 4 set_mon_vals no callback set 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.297+0000 7f6a7affd640 10 monclient: dump: 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a8c4b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a8c4b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a8c4b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6c005090 con 0x7f6a8413c010 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f6a7c003f90 con 0x7f6a8413c010 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(279..279 src has 1..279) ==== 5457+0+0 (unknown 1445761015 0 0) 0x7f6a7c04bb20 con 0x7f6a8413c010 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a7affd640 10 client.104373.objecter ms_dispatch 0x7f6a84000ba0 osd_map(279..279 src has 1..279) 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a7affd640 3 client.104373.objecter handle_osd_map got epochs [279,279] > 0 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a7affd640 3 client.104373.objecter handle_osd_map decoding full epoch 279 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a7affd640 20 client.104373.objecter dump_active .. 0 homeless 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a8a252640 10 monclient: get_auth_request con 0x7f6a6803e060 auth_method 0 2024-09-16T03:10:52.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.301+0000 7f6a7affd640 10 client.104373.objecter ms_handle_connect 0x7f6a6803e060 2024-09-16T03:10:52.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.305+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f6a7c01b0d0 con 0x7f6a8413c010 2024-09-16T03:10:52.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.305+0000 7f6a7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:52.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.305+0000 7f6a7affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:52.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.541+0000 7f6a8c4b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:10:52.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.541+0000 7f6a8c4b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:10:52.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.541+0000 7f6a8c4b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:52.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.541+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6a6c005380 con 0x7f6a8413c010 2024-09-16T03:10:52.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.541+0000 7f6a7affd640 1 -- 172.21.15.45:0/47235060 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2024881518) 0x7f6a7c020010 con 0x7f6a8413c010 2024-09-16T03:10:52.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.541+0000 7f6a7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:10:52.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.541+0000 7f6a7affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:52.546 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.541+0000 7f6a8c4b6640 10 librados: watch_flush enter 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 10 librados: watch_flush exit 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 20 client.104373.objecter shutdown clearing up homeless session... 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 10 client.104373.objecter successfully canceled tick 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 >> v1:172.21.15.46:6812/149360280 conn(0x7f6a6803e060 legacy=0x7f6a68040520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 10 monclient: shutdown 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 >> v1:172.21.15.45:6789/0 conn(0x7f6a8413c010 legacy=0x7f6a841d4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 shutdown_connections 2024-09-16T03:10:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 >> 172.21.15.45:0/47235060 conn(0x7f6a840a2bd0 msgr2=0x7f6a8413ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:52.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 shutdown_connections 2024-09-16T03:10:52.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 1 -- 172.21.15.45:0/47235060 wait complete. 2024-09-16T03:10:52.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:52.545+0000 7f6a8c4b6640 1 librados: shutdown 2024-09-16T03:10:55.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:10:55.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.693+0000 7fdc9edfe640 10 monclient: get_monmap_and_config 2024-09-16T03:10:55.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.693+0000 7fdc9edfe640 10 monclient: build_initial_monmap 2024-09-16T03:10:55.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.693+0000 7fdc9edfe640 10 monclient: monmap: 2024-09-16T03:10:55.700 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:55.700 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:55.700 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:55.699909+0000 2024-09-16T03:10:55.700 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:55.699909+0000 2024-09-16T03:10:55.701 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:55.701 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:55.701 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:55.701 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:55.701 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:55.701 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:55.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 Processor -- start 2024-09-16T03:10:55.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 -- start start 2024-09-16T03:10:55.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient: init 2024-09-16T03:10:55.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:55.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:10:55.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient(hunting): picked mon.noname-b con 0x7fdc980a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:55.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient(hunting): picked mon.noname-a con 0x7fdc980a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:55.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient(hunting): picked mon.noname-c con 0x7fdc98073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:55.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc9809d9e0 con 0x7fdc980a5810 2024-09-16T03:10:55.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc9809dc20 con 0x7fdc98073d40 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc98073360 con 0x7fdc980a4d20 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient(hunting): authenticate will time out at 1346.349313s 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9cb9a640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fdc980a4d20 legacy=0x7fdc980a5100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc97fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdc980a5810 0x7fdc9813cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58448/0 (socket says 172.21.15.45:58448) 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc97fff640 1 -- 172.21.15.45:0/1686243612 learned_addr learned my addr 172.21.15.45:0/1686243612 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 -- 172.21.15.45:0/1686243612 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150029825 0 0) 0x7fdc9809d9e0 con 0x7fdc980a5810 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient(hunting): _negotiate 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient(hunting): my global_id is 104376 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 -- 172.21.15.45:0/1686243612 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc84003530 con 0x7fdc980a5810 2024-09-16T03:10:55.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 -- 172.21.15.45:0/1686243612 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1128779913 0 0) 0x7fdc84003530 con 0x7fdc980a5810 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 -- 172.21.15.45:0/1686243612 >> v1:172.21.15.45:6790/0 conn(0x7fdc980a4d20 legacy=0x7fdc980a5100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 -- 172.21.15.45:0/1686243612 >> v1:172.21.15.46:6789/0 conn(0x7fdc98073d40 legacy=0x7fdc98140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: _finish_hunting 0 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 monclient: found mon.noname-a 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 -- 172.21.15.45:0/1686243612 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9809cf30 con 0x7fdc980a5810 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: _finish_auth 0 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: _check_auth_tickets 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 -- 172.21.15.45:0/1686243612 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdc880016a0 con 0x7fdc980a5810 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 5 monclient: authenticate success, global_id 104376 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:55.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: dump: 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 -- 172.21.15.45:0/1686243612 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc880018c0 con 0x7fdc980a5810 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient: get_monmap_and_config success 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 4 set_mon_vals no callback set 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 1 -- 172.21.15.45:0/1686243612 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdc88001d50 con 0x7fdc980a5810 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc977fe640 10 monclient: dump: 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient: shutdown 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 -- 172.21.15.45:0/1686243612 >> v1:172.21.15.45:6789/0 conn(0x7fdc980a5810 legacy=0x7fdc9813cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 -- 172.21.15.45:0/1686243612 shutdown_connections 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 -- 172.21.15.45:0/1686243612 >> 172.21.15.45:0/1686243612 conn(0x7fdc980aa720 msgr2=0x7fdc980aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 -- 172.21.15.45:0/1686243612 shutdown_connections 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 -- 172.21.15.45:0/1686243612 wait complete. 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient: build_initial_monmap 2024-09-16T03:10:55.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 10 monclient: monmap: 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 librados: starting msgr at 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.697+0000 7fdc9edfe640 1 librados: starting objecter 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 Processor -- start 2024-09-16T03:10:55.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 -- start start 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 librados: setting wanted keys 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 librados: calling monclient init 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient: init 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient(hunting): picked mon.noname-a con 0x7fdc98073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient(hunting): picked mon.noname-b con 0x7fdc980a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient(hunting): picked mon.noname-c con 0x7fdc980a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc9813f180 con 0x7fdc98073d40 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc980a1cd0 con 0x7fdc980a4d20 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc980a1510 con 0x7fdc980a5810 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:55.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient(hunting): authenticate will time out at 1346.352834s 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9d39b640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fdc980a5810 legacy=0x7fdc981d58e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9cb9a640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdc98073d40 0x7fdc981cd1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58454/0 (socket says 172.21.15.45:58454) 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9cb9a640 1 -- 172.21.15.45:0/3173466795 learned_addr learned my addr 172.21.15.45:0/3173466795 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 client.?.objecter ms_handle_connect 0x7fdc98073d40 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3493584278 0 0) 0x7fdc9813f180 con 0x7fdc98073d40 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient(hunting): _negotiate 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient(hunting): my global_id is 104379 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc78004ec0 con 0x7fdc98073d40 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 client.?.objecter ms_handle_connect 0x7fdc980a4d20 2024-09-16T03:10:55.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:10:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 649097528 0 0) 0x7fdc78004ec0 con 0x7fdc98073d40 2024-09-16T03:10:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc9813f180 con 0x7fdc98073d40 2024-09-16T03:10:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdc800041e0 con 0x7fdc98073d40 2024-09-16T03:10:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:10:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:10:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient(hunting): dump: 2024-09-16T03:10:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4251012562 0 0) 0x7fdc9813f180 con 0x7fdc98073d40 2024-09-16T03:10:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 >> v1:172.21.15.45:6790/0 conn(0x7fdc980a5810 legacy=0x7fdc981d58e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 >> v1:172.21.15.46:6789/0 conn(0x7fdc980a4d20 legacy=0x7fdc981d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient: _finish_hunting 0 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 monclient: found mon.a 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9813e1e0 con 0x7fdc98073d40 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient: _finish_auth 0 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient: _check_auth_tickets 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 5 monclient: authenticate success, global_id 104379 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient: _renew_subs 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:55.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc98140260 con 0x7fdc98073d40 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 client.104379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient: _renew_subs 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc98138b10 con 0x7fdc98073d40 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 librados: init done 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc80004380 con 0x7fdc98073d40 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdc80004790 con 0x7fdc98073d40 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc96ffd640 4 set_mon_vals no callback set 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 monclient: dump: 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:10:55.720 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc7c005090 con 0x7fdc98073d40 2024-09-16T03:10:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fdc80004930 con 0x7fdc98073d40 2024-09-16T03:10:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(279..279 src has 1..279) ==== 5457+0+0 (unknown 1445761015 0 0) 0x7fdc8004b7f0 con 0x7fdc98073d40 2024-09-16T03:10:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 client.104379.objecter ms_dispatch 0x7fdc98000ba0 osd_map(279..279 src has 1..279) 2024-09-16T03:10:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 3 client.104379.objecter handle_osd_map got epochs [279,279] > 0 2024-09-16T03:10:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 3 client.104379.objecter handle_osd_map decoding full epoch 279 2024-09-16T03:10:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 20 client.104379.objecter dump_active .. 0 homeless 2024-09-16T03:10:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc97fff640 10 monclient: get_auth_request con 0x7fdc7803df70 auth_method 0 2024-09-16T03:10:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.701+0000 7fdc957fa640 10 client.104379.objecter ms_handle_connect 0x7fdc7803df70 2024-09-16T03:10:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.705+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fdc80016a20 con 0x7fdc98073d40 2024-09-16T03:10:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.705+0000 7fdc957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:10:55.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.705+0000 7fdc957fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:10:55.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.945+0000 7fdc9edfe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:10:55.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.945+0000 7fdc9edfe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:10:55.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.945+0000 7fdc9edfe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:10:55.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.945+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdc7c005380 con 0x7fdc98073d40 2024-09-16T03:10:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.945+0000 7fdc957fa640 1 -- 172.21.15.45:0/3173466795 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 1202948206) 0x7fdc80016320 con 0x7fdc98073d40 2024-09-16T03:10:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.945+0000 7fdc957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:10:55.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.945+0000 7fdc957fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:10:55.950 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:10:55.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 10 librados: watch_flush enter 2024-09-16T03:10:55.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 10 librados: watch_flush exit 2024-09-16T03:10:55.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 20 client.104379.objecter shutdown clearing up homeless session... 2024-09-16T03:10:55.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 10 client.104379.objecter successfully canceled tick 2024-09-16T03:10:55.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 >> v1:172.21.15.46:6812/149360280 conn(0x7fdc7803df70 legacy=0x7fdc78040430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:55.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 10 monclient: shutdown 2024-09-16T03:10:55.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:10:55.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 >> v1:172.21.15.45:6789/0 conn(0x7fdc98073d40 legacy=0x7fdc981cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:10:55.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 shutdown_connections 2024-09-16T03:10:55.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 >> 172.21.15.45:0/3173466795 conn(0x7fdc980aa720 msgr2=0x7fdc980a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:10:55.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 shutdown_connections 2024-09-16T03:10:55.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 1 -- 172.21.15.45:0/3173466795 wait complete. 2024-09-16T03:10:55.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:55.949+0000 7fdc9edfe640 1 librados: shutdown 2024-09-16T03:10:58.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:10:59.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.097+0000 7f116b2a2640 10 monclient: get_monmap_and_config 2024-09-16T03:10:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.097+0000 7f116b2a2640 10 monclient: build_initial_monmap 2024-09-16T03:10:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.097+0000 7f116b2a2640 10 monclient: monmap: 2024-09-16T03:10:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:10:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:10:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:10:59.102363+0000 2024-09-16T03:10:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:10:59.102363+0000 2024-09-16T03:10:59.104 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:10:59.104 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:10:59.104 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:10:59.104 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:10:59.104 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:10:59.104 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:10:59.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.097+0000 7f116b2a2640 1 Processor -- start 2024-09-16T03:10:59.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.097+0000 7f116b2a2640 1 -- start start 2024-09-16T03:10:59.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.097+0000 7f116b2a2640 10 monclient: init 2024-09-16T03:10:59.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 10 monclient: _reopen_session rank -1 2024-09-16T03:10:59.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f1164137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 10 monclient(hunting): picked mon.noname-c con 0x7f1164138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 10 monclient(hunting): picked mon.noname-a con 0x7f116413e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11640a4f90 con 0x7f116413e2a0 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11640a4cb0 con 0x7f1164138100 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f116409dc70 con 0x7f1164137d20 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 10 monclient(hunting): _renew_subs 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116b2a2640 10 monclient(hunting): authenticate will time out at 1349.752521s 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116903e640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f11640a5c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116983f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f116413e2a0 0x7f1164140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58458/0 (socket says 172.21.15.45:58458) 2024-09-16T03:10:59.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.101+0000 7f116983f640 1 -- 172.21.15.45:0/959033241 learned_addr learned my addr 172.21.15.45:0/959033241 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:10:59.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.305+0000 7f116903e640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f11640a5c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:10:59.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:10:59.709+0000 7f116903e640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f11640a5c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:00.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:00.513+0000 7f116903e640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f11640a5c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:02.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 10 monclient(hunting): tick 2024-09-16T03:11:02.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:11:02.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 1 monclient(hunting): continuing hunt 2024-09-16T03:11:02.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:11:02.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f11640a5c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:11:02.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.46:6789/0 conn(0x7f1164138100 legacy=0x7f116413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6789/0 conn(0x7f116413e2a0 legacy=0x7f1164140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 10 monclient(hunting): picked mon.noname-a con 0x7f11480036d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 10 monclient(hunting): picked mon.noname-c con 0x7f11480072c0 addr v1:172.21.15.46:6789/0 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 10 monclient(hunting): picked mon.noname-b con 0x7f1164137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f116409cc50 con 0x7f11480036d0 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f116409dc70 con 0x7f11480072c0 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1148010910 con 0x7f1164137d20 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f115affd640 10 monclient(hunting): _renew_subs 2024-09-16T03:11:02.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.101+0000 7f116983f640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.305+0000 7f116983f640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:02.709+0000 7f116983f640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:03.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:03.513+0000 7f116983f640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:05.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 10 monclient(hunting): tick 2024-09-16T03:11:05.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:11:05.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 1 monclient(hunting): continuing hunt 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.46:6789/0 conn(0x7f11480072c0 legacy=0x7f1148009750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6789/0 conn(0x7f11480036d0 legacy=0x7f1148005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 10 monclient(hunting): picked mon.noname-c con 0x7f1148010be0 addr v1:172.21.15.46:6789/0 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 10 monclient(hunting): picked mon.noname-b con 0x7f1164137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 10 monclient(hunting): picked mon.noname-a con 0x7f114800d7f0 addr v1:172.21.15.45:6789/0 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1148011b90 con 0x7f114800d7f0 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1148010910 con 0x7f1148010be0 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 1 -- 172.21.15.45:0/959033241 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f114800e380 con 0x7f1164137d20 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f115affd640 10 monclient(hunting): _renew_subs 2024-09-16T03:11:05.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.101+0000 7f116883d640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.305+0000 7f116883d640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:05.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:05.709+0000 7f116883d640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:06.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:06.513+0000 7f116883d640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:07.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.065+0000 7f115bfff640 1 -- 172.21.15.45:0/959033241 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 483052407 0 0) 0x7f1148011b90 con 0x7f114800d7f0 2024-09-16T03:11:07.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.065+0000 7f115bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.065+0000 7f115bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.065+0000 7f115bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.065+0000 7f115bfff640 10 monclient(hunting): my global_id is 104382 2024-09-16T03:11:07.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.065+0000 7f115bfff640 1 -- 172.21.15.45:0/959033241 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1150003530 con 0x7f114800d7f0 2024-09-16T03:11:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 1 -- 172.21.15.45:0/959033241 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3117713014 0 0) 0x7f1150003530 con 0x7f114800d7f0 2024-09-16T03:11:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6790/0 conn(0x7f1164137d20 legacy=0x7f114800d0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:11:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.46:6789/0 conn(0x7f1148010be0 legacy=0x7f1148011020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:11:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 1 monclient: found mon.noname-a 2024-09-16T03:11:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:11:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 1 -- 172.21.15.45:0/959033241 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114800e550 con 0x7f114800d7f0 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: _finish_auth 0 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 1 -- 172.21.15.45:0/959033241 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f115c0019a0 con 0x7f114800d7f0 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: dump: 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.074 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.075 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.075 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.075 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f116b2a2640 5 monclient: authenticate success, global_id 104382 2024-09-16T03:11:07.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f116b2a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:11:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 1 -- 172.21.15.45:0/959033241 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f115c001b40 con 0x7f114800d7f0 2024-09-16T03:11:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:11:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 1 -- 172.21.15.45:0/959033241 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f115c001f90 con 0x7f114800d7f0 2024-09-16T03:11:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:11:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f115bfff640 10 monclient: dump: 2024-09-16T03:11:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.076 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.077 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.077 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.077 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.077 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.077 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.077 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.077 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f116b2a2640 10 monclient: get_monmap_and_config success 2024-09-16T03:11:07.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.069+0000 7f116b2a2640 4 set_mon_vals no callback set 2024-09-16T03:11:07.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 10 monclient: shutdown 2024-09-16T03:11:07.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:11:07.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 1 -- 172.21.15.45:0/959033241 >> v1:172.21.15.45:6789/0 conn(0x7f114800d7f0 legacy=0x7f114800dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 1 -- 172.21.15.45:0/959033241 shutdown_connections 2024-09-16T03:11:07.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 1 -- 172.21.15.45:0/959033241 >> 172.21.15.45:0/959033241 conn(0x7f11640aa110 msgr2=0x7f11640aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:11:07.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 1 -- 172.21.15.45:0/959033241 shutdown_connections 2024-09-16T03:11:07.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 1 -- 172.21.15.45:0/959033241 wait complete. 2024-09-16T03:11:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 10 monclient: build_initial_monmap 2024-09-16T03:11:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 10 monclient: monmap: 2024-09-16T03:11:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:11:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:11:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:11:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:11:07.079 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:11:07.080 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.080 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:11:07.080 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:11:07.080 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:11:07.080 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 1 librados: starting msgr at 2024-09-16T03:11:07.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 1 librados: starting objecter 2024-09-16T03:11:07.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.073+0000 7f116b2a2640 1 Processor -- start 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 1 -- start start 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 1 librados: setting wanted keys 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 1 librados: calling monclient init 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 10 monclient: init 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 10 monclient: _reopen_session rank -1 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 10 monclient(hunting): picked mon.noname-c con 0x7f114800d7f0 addr v1:172.21.15.45:6790/0 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f1148010be0 addr v1:172.21.15.46:6789/0 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116903e640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f114800d7f0 legacy=0x7f1164139100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 10 monclient(hunting): picked mon.noname-a con 0x7f1164137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f116413cf20 con 0x7f1164137d20 2024-09-16T03:11:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1164139f20 con 0x7f1148010be0 2024-09-16T03:11:07.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11640a1300 con 0x7f114800d7f0 2024-09-16T03:11:07.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 10 monclient(hunting): _renew_subs 2024-09-16T03:11:07.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116b2a2640 10 monclient(hunting): authenticate will time out at 1357.729527s 2024-09-16T03:11:07.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116983f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1164137d20 0x7f11641d5480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52050/0 (socket says 172.21.15.45:52050) 2024-09-16T03:11:07.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f116983f640 1 -- 172.21.15.45:0/3281183266 learned_addr learned my addr 172.21.15.45:0/3281183266 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:11:07.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 client.?.objecter ms_handle_connect 0x7f1164137d20 2024-09-16T03:11:07.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:11:07.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 client.?.objecter ms_handle_connect 0x7f1148010be0 2024-09-16T03:11:07.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:11:07.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3652260142 0 0) 0x7f116413cf20 con 0x7f1164137d20 2024-09-16T03:11:07.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 monclient(hunting): my global_id is 104385 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1148011b90 con 0x7f1164137d20 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1538807793 0 0) 0x7f1148011b90 con 0x7f1164137d20 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f116413cf20 con 0x7f1164137d20 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f115c001e90 con 0x7f1164137d20 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 monclient(hunting): dump: 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.087 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.088 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.088 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.088 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:11:07.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 410194634 0 0) 0x7f116413cf20 con 0x7f1164137d20 2024-09-16T03:11:07.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 >> v1:172.21.15.45:6790/0 conn(0x7f114800d7f0 legacy=0x7f1164139100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:11:07.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 >> v1:172.21.15.46:6789/0 conn(0x7f1148010be0 legacy=0x7f1164139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.077+0000 7f1159ffb640 1 monclient: found mon.a 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1164074a40 con 0x7f1164137d20 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 10 monclient: _finish_auth 0 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 5 monclient: authenticate success, global_id 104385 2024-09-16T03:11:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 10 monclient: _renew_subs 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f116413abc0 con 0x7f1164137d20 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f115c004e20 con 0x7f1164137d20 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f115c005170 con 0x7f1164137d20 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 10 monclient: dump: 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.090 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f115b7fe640 4 set_mon_vals no callback set 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 10 client.104385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 10 monclient: _renew_subs 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1164134700 con 0x7f1164137d20 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 1 librados: init done 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f115c0111e0 con 0x7f1164137d20 2024-09-16T03:11:07.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(279..279 src has 1..279) ==== 5457+0+0 (unknown 1445761015 0 0) 0x7f115c04ba20 con 0x7f1164137d20 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 10 client.104385.objecter ms_dispatch 0x7f1164000ba0 osd_map(279..279 src has 1..279) 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 3 client.104385.objecter handle_osd_map got epochs [279,279] > 0 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 3 client.104385.objecter handle_osd_map decoding full epoch 279 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 20 client.104385.objecter dump_active .. 0 homeless 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11641373f0 con 0x7f1164137d20 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f116903e640 10 monclient: get_auth_request con 0x7f114803f6f0 auth_method 0 2024-09-16T03:11:07.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.081+0000 7f1159ffb640 10 client.104385.objecter ms_handle_connect 0x7f114803f6f0 2024-09-16T03:11:07.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.085+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f115c016d10 con 0x7f1164137d20 2024-09-16T03:11:07.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.085+0000 7f1159ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:11:07.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.085+0000 7f1159ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:11:07.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f116b2a2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:11:07.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f116b2a2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:11:07.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f116b2a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f11640767c0 con 0x7f1164137d20 2024-09-16T03:11:07.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f1159ffb640 1 -- 172.21.15.45:0/3281183266 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3604371482) 0x7f115c016610 con 0x7f1164137d20 2024-09-16T03:11:07.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f1159ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:11:07.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f1159ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:11:07.330 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:11:07.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f116b2a2640 10 librados: watch_flush enter 2024-09-16T03:11:07.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f116b2a2640 10 librados: watch_flush exit 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f116b2a2640 20 client.104385.objecter shutdown clearing up homeless session... 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.325+0000 7f116b2a2640 10 client.104385.objecter successfully canceled tick 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.329+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 >> v1:172.21.15.46:6812/149360280 conn(0x7f114803f6f0 legacy=0x7f1148041bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.329+0000 7f116b2a2640 10 monclient: shutdown 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.329+0000 7f116b2a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.329+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 >> v1:172.21.15.45:6789/0 conn(0x7f1164137d20 legacy=0x7f11641d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.329+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 shutdown_connections 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.329+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 >> 172.21.15.45:0/3281183266 conn(0x7f11640aa110 msgr2=0x7f1164073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.329+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 shutdown_connections 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.329+0000 7f116b2a2640 1 -- 172.21.15.45:0/3281183266 wait complete. 2024-09-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.329+0000 7f116b2a2640 1 librados: shutdown 2024-09-16T03:11:07.349 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:11:07.349 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:11:07.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:11:07.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient: get_monmap_and_config 2024-09-16T03:11:07.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient: build_initial_monmap 2024-09-16T03:11:07.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient: monmap: 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:11:07.473604+0000 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:11:07.473604+0000 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:11:07.474 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 1 Processor -- start 2024-09-16T03:11:07.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 1 -- start start 2024-09-16T03:11:07.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient: init 2024-09-16T03:11:07.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient: _reopen_session rank -1 2024-09-16T03:11:07.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient(hunting): picked mon.noname-c con 0x7f0f3c135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient(hunting): picked mon.noname-a con 0x7f0f3c09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient(hunting): picked mon.noname-b con 0x7f0f3c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f3c09f7e0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f3c09b160 con 0x7f0f3c135950 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f3c09b440 con 0x7f0f3c13c010 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient(hunting): _renew_subs 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient(hunting): authenticate will time out at 1358.122901s 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f41af7640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f0f3c13c010 legacy=0x7f0f3c13e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f40af5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0f3c09fc40 0x7f0f3c13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52058/0 (socket says 172.21.15.45:52058) 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f40af5640 1 -- 172.21.15.45:0/118422702 learned_addr learned my addr 172.21.15.45:0/118422702 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2239029231 0 0) 0x7f0f3c09f7e0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient(hunting): my global_id is 104388 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f28003530 con 0x7f0f3c09fc40 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090475783 0 0) 0x7f0f3c09b160 con 0x7f0f3c135950 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient(hunting): my global_id is 22876 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f3c09f7e0 con 0x7f0f3c135950 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 840969561 0 0) 0x7f0f28003530 con 0x7f0f3c09fc40 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 >> v1:172.21.15.45:6790/0 conn(0x7f0f3c13c010 legacy=0x7f0f3c13e400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 >> v1:172.21.15.46:6789/0 conn(0x7f0f3c135950 legacy=0x7f0f3c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: _finish_hunting 0 2024-09-16T03:11:07.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 monclient: found mon.noname-a 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3c136440 con 0x7f0f3c09fc40 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: _finish_auth 0 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: _check_auth_tickets 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 5 monclient: authenticate success, global_id 104388 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f2c0016a0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: dump: 2024-09-16T03:11:07.480 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0f2c001840 con 0x7f0f3c09fc40 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:11:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 1 -- 172.21.15.45:0/118422702 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f2c001c90 con 0x7f0f3c09fc40 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 10 monclient: get_monmap_and_config success 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f4355a640 4 set_mon_vals no callback set 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.469+0000 7f0f33fff640 10 monclient: dump: 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: shutdown 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- 172.21.15.45:0/118422702 >> v1:172.21.15.45:6789/0 conn(0x7f0f3c09fc40 legacy=0x7f0f3c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- 172.21.15.45:0/118422702 shutdown_connections 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- 172.21.15.45:0/118422702 >> 172.21.15.45:0/118422702 conn(0x7f0f3c0a2bd0 msgr2=0x7f0f3c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- 172.21.15.45:0/118422702 shutdown_connections 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- 172.21.15.45:0/118422702 wait complete. 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: build_initial_monmap 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: monmap: 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:11:07.483 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 librados: starting msgr at 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 librados: starting objecter 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 Processor -- start 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- start start 2024-09-16T03:11:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 librados: setting wanted keys 2024-09-16T03:11:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 librados: calling monclient init 2024-09-16T03:11:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: init 2024-09-16T03:11:07.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: _reopen_session rank -1 2024-09-16T03:11:07.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:11:07.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient(hunting): picked mon.noname-a con 0x7f0f3c09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:11:07.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient(hunting): picked mon.noname-b con 0x7f0f3c135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:11:07.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient(hunting): picked mon.noname-c con 0x7f0f3c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f3c13bcb0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f3c13b780 con 0x7f0f3c135950 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f3c09d2c0 con 0x7f0f3c13c010 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient(hunting): _renew_subs 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient(hunting): authenticate will time out at 1358.126231s 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f41af7640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f0f3c13c010 legacy=0x7f0f3c1d4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f412f6640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0f3c09fc40 0x7f0f3c1ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52060/0 (socket says 172.21.15.45:52060) 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f412f6640 1 -- 172.21.15.45:0/213606999 learned_addr learned my addr 172.21.15.45:0/213606999 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 client.?.objecter ms_handle_connect 0x7f0f3c09fc40 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:11:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 client.?.objecter ms_handle_connect 0x7f0f3c135950 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881905909 0 0) 0x7f0f3c13bcb0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): my global_id is 104391 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f20004ec0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367847879 0 0) 0x7f0f3c13b780 con 0x7f0f3c135950 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): my global_id is 22879 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f3c13bcb0 con 0x7f0f3c135950 2024-09-16T03:11:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4140072001 0 0) 0x7f0f20004ec0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f3c13b780 con 0x7f0f3c09fc40 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f38001ac0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): dump: 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1897495020 0 0) 0x7f0f3c13b780 con 0x7f0f3c09fc40 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 >> v1:172.21.15.45:6790/0 conn(0x7f0f3c13c010 legacy=0x7f0f3c1d4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 >> v1:172.21.15.46:6789/0 conn(0x7f0f3c135950 legacy=0x7f0f3c0737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 monclient: found mon.a 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3c09d7d0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient: _finish_auth 0 2024-09-16T03:11:07.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 5 monclient: authenticate success, global_id 104391 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: _renew_subs 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0f3c1cd4c0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 client.104391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: _renew_subs 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0f3c1d64d0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f4355a640 1 librados: init done 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0f38001c60 con 0x7f0f3c09fc40 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:11:07.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f38003af0 con 0x7f0f3c09fc40 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f337fe640 4 set_mon_vals no callback set 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.473+0000 7f0f31ffb640 10 monclient: dump: 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.499 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f4355a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:11:07.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0f1c0065d0 con 0x7f0f1c002870 2024-09-16T03:11:07.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 client.104391.objecter ms_handle_connect 0x7f0f1c002870 2024-09-16T03:11:07.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 client.104391.objecter resend_mon_ops 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f0f38003d10 con 0x7f0f3c09fc40 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(279..279 src has 1..279) ==== 5457+0+0 (unknown 1445761015 0 0) 0x7f0f3804b830 con 0x7f0f3c09fc40 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 client.104391.objecter ms_dispatch 0x7f0f3c000ba0 osd_map(279..279 src has 1..279) 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 3 client.104391.objecter handle_osd_map got epochs [279,279] > 0 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 3 client.104391.objecter handle_osd_map decoding full epoch 279 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 20 client.104391.objecter dump_active .. 0 homeless 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1041421409 0 0) 0x7f0f1c0065d0 con 0x7f0f1c002870 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 monclient(hunting): my global_id is 104394 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f3c13b780 con 0x7f0f1c002870 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 619013213 0 0) 0x7f0f3c13b780 con 0x7f0f1c002870 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f1c0065d0 con 0x7f0f1c002870 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f41af7640 10 monclient: get_auth_request con 0x7f0f2003df90 auth_method 0 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f2c001910 con 0x7f0f1c002870 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 monclient: dump: 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.502 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 208648770 0 0) 0x7f0f1c0065d0 con 0x7f0f1c002870 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f1c00a170 con 0x7f0f1c002870 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 client.104391.objecter ms_handle_connect 0x7f0f2003df90 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0f2c001a20 con 0x7f0f1c002870 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 >> v1:172.21.15.45:6789/0 conn(0x7f0f1c002870 legacy=0x7f0f1c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.477+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 >> v1:172.21.15.45:6789/0 conn(0x7f0f1c002870 legacy=0x7f0f1c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.481+0000 7f0f4355a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.481+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0f1c009e80 con 0x7f0f1c009140 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.481+0000 7f0f31ffb640 10 client.104391.objecter ms_handle_connect 0x7f0f1c009140 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.481+0000 7f0f31ffb640 10 client.104391.objecter resend_mon_ops 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865299708 0 0) 0x7f0f1c009e80 con 0x7f0f1c009140 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 10 monclient(hunting): my global_id is 104397 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f1c0065d0 con 0x7f0f1c009140 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 422894661 0 0) 0x7f0f1c0065d0 con 0x7f0f1c009140 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f1c009e80 con 0x7f0f1c009140 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f2c004990 con 0x7f0f1c009140 2024-09-16T03:11:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:11:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 10 monclient: dump: 2024-09-16T03:11:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2654696996 0 0) 0x7f0f1c009e80 con 0x7f0f1c009140 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0f1c00c400 con 0x7f0f1c009140 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4126 (unknown 0 0 828670093) 0x7f0f2c004920 con 0x7f0f1c009140 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 >> v1:172.21.15.45:6789/0 conn(0x7f0f1c009140 legacy=0x7f0f1c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f31ffb640 1 -- 172.21.15.45:0/213606999 >> v1:172.21.15.45:6789/0 conn(0x7f0f1c009140 legacy=0x7f0f1c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 30, 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 126586, 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: 1 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:11:07.507 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:11:07.508 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:11:07.509 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:11:07.510 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:11:07.511 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.512 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.513 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:11:07.514 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:11:07.515 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:11:07.516 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:11:07.516 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.516 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.516 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:11:07.517 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:11:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.519 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:11:07.519 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:11:07.519 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:11:07.519 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.519 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.519 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:11:07.519 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 10 librados: watch_flush enter 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 10 librados: watch_flush exit 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 20 client.104391.objecter shutdown clearing up homeless session... 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 10 client.104391.objecter successfully canceled tick 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 >> v1:172.21.15.46:6812/149360280 conn(0x7f0f2003df90 legacy=0x7f0f20040450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 10 monclient: shutdown 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 >> v1:172.21.15.45:6789/0 conn(0x7f0f3c09fc40 legacy=0x7f0f3c1ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f41af7640 1 -- 172.21.15.45:0/213606999 reap_dead start 2024-09-16T03:11:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f41af7640 1 -- 172.21.15.45:0/213606999 reap_dead start 2024-09-16T03:11:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 shutdown_connections 2024-09-16T03:11:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 >> 172.21.15.45:0/213606999 conn(0x7f0f3c0a2bd0 msgr2=0x7f0f3c13eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:11:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 shutdown_connections 2024-09-16T03:11:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 1 -- 172.21.15.45:0/213606999 wait complete. 2024-09-16T03:11:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.485+0000 7f0f4355a640 1 librados: shutdown 2024-09-16T03:11:07.522 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:11:07.522 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:11:07.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:11:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient: get_monmap_and_config 2024-09-16T03:11:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient: build_initial_monmap 2024-09-16T03:11:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient: monmap: 2024-09-16T03:11:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:11:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:11:07.641142+0000 2024-09-16T03:11:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:11:07.641142+0000 2024-09-16T03:11:07.641 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:11:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:11:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:11:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:11:07.642 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 1 Processor -- start 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 1 -- start start 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient: init 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient: _reopen_session rank -1 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient(hunting): picked mon.noname-a con 0x7f9e4c137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient(hunting): picked mon.noname-c con 0x7f9e4c138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient(hunting): picked mon.noname-b con 0x7f9e4c13e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c0a4f90 con 0x7f9e4c137d20 2024-09-16T03:11:07.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c0a4cb0 con 0x7f9e4c138100 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c09dc70 con 0x7f9e4c13e2a0 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient(hunting): _renew_subs 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient(hunting): authenticate will time out at 1358.290548s 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e519fe640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f9e4c13e2a0 legacy=0x7f9e4c140690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e511fd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9e4c137d20 0x7f9e4c0a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52094/0 (socket says 172.21.15.45:52094) 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e511fd640 1 -- 172.21.15.45:0/3258411301 learned_addr learned my addr 172.21.15.45:0/3258411301 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 -- 172.21.15.45:0/3258411301 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92311170 0 0) 0x7f9e4c0a4f90 con 0x7f9e4c137d20 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient(hunting): my global_id is 104400 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 -- 172.21.15.45:0/3258411301 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e38003530 con 0x7f9e4c137d20 2024-09-16T03:11:07.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 -- 172.21.15.45:0/3258411301 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 77325296 0 0) 0x7f9e38003530 con 0x7f9e4c137d20 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 -- 172.21.15.45:0/3258411301 >> v1:172.21.15.45:6790/0 conn(0x7f9e4c13e2a0 legacy=0x7f9e4c140690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 -- 172.21.15.45:0/3258411301 >> v1:172.21.15.46:6789/0 conn(0x7f9e4c138100 legacy=0x7f9e4c13cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: _finish_hunting 0 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 monclient: found mon.noname-a 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 -- 172.21.15.45:0/3258411301 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4c09cc50 con 0x7f9e4c137d20 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: _finish_auth 0 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: _check_auth_tickets 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 -- 172.21.15.45:0/3258411301 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9e480016a0 con 0x7f9e4c137d20 2024-09-16T03:11:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: dump: 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 5 monclient: authenticate success, global_id 104400 2024-09-16T03:11:07.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 -- 172.21.15.45:0/3258411301 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9e48001840 con 0x7f9e4c137d20 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 10 monclient: get_monmap_and_config success 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e53461640 4 set_mon_vals no callback set 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 1 -- 172.21.15.45:0/3258411301 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9e48001cd0 con 0x7f9e4c137d20 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.637+0000 7f9e43fff640 10 monclient: dump: 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.654 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: shutdown 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- 172.21.15.45:0/3258411301 >> v1:172.21.15.45:6789/0 conn(0x7f9e4c137d20 legacy=0x7f9e4c0a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- 172.21.15.45:0/3258411301 shutdown_connections 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- 172.21.15.45:0/3258411301 >> 172.21.15.45:0/3258411301 conn(0x7f9e4c0aa110 msgr2=0x7f9e4c0aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- 172.21.15.45:0/3258411301 shutdown_connections 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- 172.21.15.45:0/3258411301 wait complete. 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: build_initial_monmap 2024-09-16T03:11:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: monmap: 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 librados: starting msgr at 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 librados: starting objecter 2024-09-16T03:11:07.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 Processor -- start 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- start start 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 librados: setting wanted keys 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 librados: calling monclient init 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: init 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: _reopen_session rank -1 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient(hunting): picked mon.noname-a con 0x7f9e4c137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient(hunting): picked mon.noname-c con 0x7f9e4c138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient(hunting): picked mon.noname-b con 0x7f9e4c13e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c139770 con 0x7f9e4c137d20 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c0a1300 con 0x7f9e4c13e2a0 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c13b6c0 con 0x7f9e4c138100 2024-09-16T03:11:07.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient(hunting): _renew_subs 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient(hunting): authenticate will time out at 1358.293698s 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e509fc640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f9e4c138100 legacy=0x7f9e4c139060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e511fd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9e4c137d20 0x7f9e4c138950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52106/0 (socket says 172.21.15.45:52106) 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e511fd640 1 -- 172.21.15.45:0/3059802896 learned_addr learned my addr 172.21.15.45:0/3059802896 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 client.?.objecter ms_handle_connect 0x7f9e4c137d20 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475215022 0 0) 0x7f9e4c139770 con 0x7f9e4c137d20 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): my global_id is 104403 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e30004ec0 con 0x7f9e4c137d20 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 client.?.objecter ms_handle_connect 0x7f9e4c13e2a0 2024-09-16T03:11:07.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2593028818 0 0) 0x7f9e30004ec0 con 0x7f9e4c137d20 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e4c139770 con 0x7f9e4c137d20 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9e48001c50 con 0x7f9e4c137d20 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): dump: 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.659 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3048512194 0 0) 0x7f9e4c0a1300 con 0x7f9e4c13e2a0 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient(hunting): my global_id is 22885 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e30004ec0 con 0x7f9e4c13e2a0 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3722505521 0 0) 0x7f9e4c139770 con 0x7f9e4c137d20 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 >> v1:172.21.15.45:6790/0 conn(0x7f9e4c138100 legacy=0x7f9e4c139060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 >> v1:172.21.15.46:6789/0 conn(0x7f9e4c13e2a0 legacy=0x7f9e4c1d4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 monclient: found mon.a 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4c13b9a0 con 0x7f9e4c137d20 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient: _finish_auth 0 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 5 monclient: authenticate success, global_id 104403 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: _renew_subs 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9e4c13a9c0 con 0x7f9e4c137d20 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 client.104403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: _renew_subs 2024-09-16T03:11:07.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9e4c13c550 con 0x7f9e4c137d20 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 librados: init done 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9e480017c0 con 0x7f9e4c137d20 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9e48004c40 con 0x7f9e4c137d20 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e437fe640 4 set_mon_vals no callback set 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 monclient: dump: 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.662 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9e2c0065d0 con 0x7f9e2c002870 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f9e48004e60 con 0x7f9e4c137d20 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(279..279 src has 1..279) ==== 5457+0+0 (unknown 1445761015 0 0) 0x7f9e4804b880 con 0x7f9e4c137d20 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 client.104403.objecter ms_dispatch 0x7f9e4c000ba0 osd_map(279..279 src has 1..279) 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 3 client.104403.objecter handle_osd_map got epochs [279,279] > 0 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 3 client.104403.objecter handle_osd_map decoding full epoch 279 2024-09-16T03:11:07.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 20 client.104403.objecter dump_active .. 0 homeless 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 client.104403.objecter ms_handle_connect 0x7f9e2c002870 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.641+0000 7f9e41ffb640 10 client.104403.objecter resend_mon_ops 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1082845912 0 0) 0x7f9e2c0065d0 con 0x7f9e2c002870 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 monclient(hunting): my global_id is 22888 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e4c139770 con 0x7f9e2c002870 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e519fe640 10 monclient: get_auth_request con 0x7f9e3003dfb0 auth_method 0 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1781989321 0 0) 0x7f9e4c139770 con 0x7f9e2c002870 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e2c0065d0 con 0x7f9e2c002870 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9e3c001740 con 0x7f9e2c002870 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 monclient: dump: 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 client.104403.objecter ms_handle_connect 0x7f9e3003dfb0 2024-09-16T03:11:07.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3525546130 0 0) 0x7f9e2c0065d0 con 0x7f9e2c002870 2024-09-16T03:11:07.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e2c00a170 con 0x7f9e2c002870 2024-09-16T03:11:07.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9e3c0019c0 con 0x7f9e2c002870 2024-09-16T03:11:07.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:11:07.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:11:07.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 >> v1:172.21.15.46:6789/0 conn(0x7f9e2c002870 legacy=0x7f9e2c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.645+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 >> v1:172.21.15.46:6789/0 conn(0x7f9e2c002870 legacy=0x7f9e2c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:11:07.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9e2c009e60 con 0x7f9e2c009120 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 client.104403.objecter ms_handle_connect 0x7f9e2c009120 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 client.104403.objecter resend_mon_ops 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2106711374 0 0) 0x7f9e2c009e60 con 0x7f9e2c009120 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 monclient(hunting): my global_id is 22891 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e2c0065d0 con 0x7f9e2c009120 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3695812453 0 0) 0x7f9e2c0065d0 con 0x7f9e2c009120 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e2c009e60 con 0x7f9e2c009120 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9e3c0037a0 con 0x7f9e2c009120 2024-09-16T03:11:07.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 monclient: dump: 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:11:07.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 108977838 0 0) 0x7f9e2c009e60 con 0x7f9e2c009120 2024-09-16T03:11:07.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9e2c00c400 con 0x7f9e2c009120 2024-09-16T03:11:07.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4291 (unknown 0 0 152846579) 0x7f9e3c0057d0 con 0x7f9e2c009120 2024-09-16T03:11:07.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:11:07.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:11:07.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 >> v1:172.21.15.46:6789/0 conn(0x7f9e2c009120 legacy=0x7f9e2c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e41ffb640 1 -- 172.21.15.45:0/3059802896 >> v1:172.21.15.46:6789/0 conn(0x7f9e2c009120 legacy=0x7f9e2c0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:11:07.670 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 30, 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 389463, 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: 1 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:11:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:11:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:11:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:11:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:11:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:11:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:11:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:11:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:11:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:11:07.683 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:11:07.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 10 librados: watch_flush enter 2024-09-16T03:11:07.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 10 librados: watch_flush exit 2024-09-16T03:11:07.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 20 client.104403.objecter shutdown clearing up homeless session... 2024-09-16T03:11:07.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 10 client.104403.objecter successfully canceled tick 2024-09-16T03:11:07.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 >> v1:172.21.15.46:6812/149360280 conn(0x7f9e3003dfb0 legacy=0x7f9e30040470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 10 monclient: shutdown 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 >> v1:172.21.15.45:6789/0 conn(0x7f9e4c137d20 legacy=0x7f9e4c138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e519fe640 1 -- 172.21.15.45:0/3059802896 reap_dead start 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e519fe640 1 -- 172.21.15.45:0/3059802896 reap_dead start 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 shutdown_connections 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 >> 172.21.15.45:0/3059802896 conn(0x7f9e4c0aa110 msgr2=0x7f9e4c0a7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 shutdown_connections 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 1 -- 172.21.15.45:0/3059802896 wait complete. 2024-09-16T03:11:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:11:07.653+0000 7f9e53461640 1 librados: shutdown 2024-09-16T03:11:07.685 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:11:07.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:11:10.074 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2:163:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:5d165639:::164:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:f43765fc:::165:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:b4c720e9:::166:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:e694b040:::167:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:afa38db2:::168:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:77ba9f53:::169:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:87495034:::170:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:7c96bf0e:::171:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:dbe346cc:::172:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:e943ec24:::173:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:f97a9c0c:::174:head 2024-09-16T03:11:10.075 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:6f26e74d:::175:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:4f95e106:::176:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:0e6f2f8f:::177:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:05db05f1:::178:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:38a78d66:::179:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:d095610b:::180:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:a1a9d709:::181:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:1e5d39db:::182:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:f7df4fb9:::183:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:03a7f161:::184:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:ba70721e:::185:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:28e5662d:::186:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:973d52de:::187:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:4303eb1c:::188:head 2024-09-16T03:11:10.076 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:b990b48e:::189:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:29b8165b:::190:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:3547f197:::191:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:7e260936:::192:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:1abec7b1:::193:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:10fdda93:::194:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:15817eea:::195:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:770bab57:::196:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:ed9e13e7:::197:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:71471a8f:::198:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: checking for 269:10fb1d02:::199:head 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (9054 ms) 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,0 2024-09-16T03:11:10.077 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: first is 1726456216 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,1726456218,1726456219,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,1726456218,1726456219,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,1726456218,1726456219,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,1726456218,1726456219,1726456221,1726456222,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,1726456218,1726456219,1726456221,1726456222,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,1726456218,1726456219,1726456221,1726456222,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,1726456218,1726456219,1726456221,1726456222,1726456225,1726456226,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,1726456218,1726456219,1726456221,1726456222,1726456225,1726456226,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456216,1726456218,1726456219,1726456221,1726456222,1726456225,1726456226,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456219,1726456221,1726456222,1726456225,1726456226,1726456228,1726456229,0 2024-09-16T03:11:10.078 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: first now 1726456219, trimmed 2024-09-16T03:11:10.079 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20962 ms) 2024-09-16T03:11:10.079 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-16T03:11:10.079 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: foo0 2024-09-16T03:11:10.079 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T03:11:10.079 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14032 ms) 2024-09-16T03:11:10.079 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-16T03:11:42.995 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ 2024-09-16T03:11:42.995 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2993 ms) 2024-09-16T03:11:42.995 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (22099 ms) 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6924 ms) 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6978 ms) 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7068 ms) 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6984 ms) 2024-09-16T03:11:42.996 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-16T03:11:42.997 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6964 ms) 2024-09-16T03:11:42.997 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (167642 ms total) 2024-09-16T03:11:42.997 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: 2024-09-16T03:11:42.997 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-16T03:11:42.997 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (341514 ms total) 2024-09-16T03:11:42.997 INFO:tasks.workunit.client.0.smithi046.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-16T03:12:37.686 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:12:37.687 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:12:37.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:12:37.691 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:12:37.691 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:12:37.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:12:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient: get_monmap_and_config 2024-09-16T03:12:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient: build_initial_monmap 2024-09-16T03:12:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient: monmap: 2024-09-16T03:12:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:37.825620+0000 2024-09-16T03:12:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:37.825620+0000 2024-09-16T03:12:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:37.826 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:37.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 1 Processor -- start 2024-09-16T03:12:37.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 1 -- start start 2024-09-16T03:12:37.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient: init 2024-09-16T03:12:37.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:37.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient(hunting): picked mon.noname-a con 0x7fc8ac0a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient(hunting): picked mon.noname-b con 0x7fc8ac0a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient(hunting): picked mon.noname-c con 0x7fc8ac073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ac09d9e0 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ac09dc20 con 0x7fc8ac073d40 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ac073360 con 0x7fc8ac0a5810 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8b1a6a640 10 monclient(hunting): authenticate will time out at 1448.474721s 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8ab7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc8ac0a4d20 0x7fc8ac0a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56930/0 (socket says 172.21.15.45:56930) 2024-09-16T03:12:37.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.820+0000 7fc8ab7fe640 1 -- 172.21.15.45:0/1666786531 learned_addr learned my addr 172.21.15.45:0/1666786531 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:37.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 -- 172.21.15.45:0/1666786531 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 656438700 0 0) 0x7fc8ac09dc20 con 0x7fc8ac073d40 2024-09-16T03:12:37.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:12:37.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:37.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:37.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient(hunting): my global_id is 22921 2024-09-16T03:12:37.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 -- 172.21.15.45:0/1666786531 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc898003530 con 0x7fc8ac073d40 2024-09-16T03:12:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 -- 172.21.15.45:0/1666786531 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2142785855 0 0) 0x7fc898003530 con 0x7fc8ac073d40 2024-09-16T03:12:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 -- 172.21.15.45:0/1666786531 >> v1:172.21.15.45:6790/0 conn(0x7fc8ac0a5810 legacy=0x7fc8ac13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 -- 172.21.15.45:0/1666786531 >> v1:172.21.15.45:6789/0 conn(0x7fc8ac0a4d20 legacy=0x7fc8ac0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 monclient: found mon.noname-c 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 -- 172.21.15.45:0/1666786531 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ac09cf30 con 0x7fc8ac073d40 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: _finish_auth 0 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 -- 172.21.15.45:0/1666786531 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc8a4001a20 con 0x7fc8ac073d40 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: dump: 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8b1a6a640 5 monclient: authenticate success, global_id 22921 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8b1a6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 -- 172.21.15.45:0/1666786531 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8a4001c40 con 0x7fc8ac073d40 2024-09-16T03:12:37.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:37.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 1 -- 172.21.15.45:0/1666786531 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc8a40020d0 con 0x7fc8ac073d40 2024-09-16T03:12:37.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:37.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:12:37.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8aa7fc640 10 monclient: dump: 2024-09-16T03:12:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8b1a6a640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.844+0000 7fc8b1a6a640 4 set_mon_vals no callback set 2024-09-16T03:12:37.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 10 monclient: shutdown 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/1666786531 >> v1:172.21.15.46:6789/0 conn(0x7fc8ac073d40 legacy=0x7fc8ac140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/1666786531 shutdown_connections 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/1666786531 >> 172.21.15.45:0/1666786531 conn(0x7fc8ac0aa720 msgr2=0x7fc8ac0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/1666786531 shutdown_connections 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/1666786531 wait complete. 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 10 monclient: build_initial_monmap 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 10 monclient: monmap: 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 librados: starting msgr at 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 librados: starting objecter 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 Processor -- start 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 -- start start 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 librados: setting wanted keys 2024-09-16T03:12:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 1 librados: calling monclient init 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 10 monclient: init 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 10 monclient(hunting): picked mon.noname-c con 0x7fc8ac073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.848+0000 7fc8b1a6a640 10 monclient(hunting): picked mon.noname-a con 0x7fc8ac0a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 monclient(hunting): picked mon.noname-b con 0x7fc8ac0a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ac1403f0 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ac1406d0 con 0x7fc8ac0a5810 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8ab7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc8ac073d40 0x7fc8ac13cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40576/0 (socket says 172.21.15.45:40576) 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8ab7fe640 1 -- 172.21.15.45:0/543237788 learned_addr learned my addr 172.21.15.45:0/543237788 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8aaffd640 1 --1- 172.21.15.45:0/543237788 >> v1:172.21.15.45:6789/0 conn(0x7fc8ac0a4d20 0x7fc8ac13d270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56934/0 (socket says 172.21.15.45:56934) 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 client.?.objecter ms_handle_connect 0x7fc8ac073d40 2024-09-16T03:12:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 client.?.objecter ms_handle_connect 0x7fc8ac0a4d20 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ac0a3c10 con 0x7fc8ac073d40 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 monclient(hunting): authenticate will time out at 1448.504343s 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432112612 0 0) 0x7fc8ac1403f0 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient(hunting): my global_id is 104436 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc88c004ec0 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 client.?.objecter ms_handle_connect 0x7fc8ac0a5810 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1868267438 0 0) 0x7fc88c004ec0 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8ac1403f0 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc894001920 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient(hunting): dump: 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3108815076 0 0) 0x7fc8ac1403f0 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 >> v1:172.21.15.45:6790/0 conn(0x7fc8ac073d40 legacy=0x7fc8ac13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 >> v1:172.21.15.46:6789/0 conn(0x7fc8ac0a5810 legacy=0x7fc8ac1d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 monclient: found mon.a 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ac0a1510 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient: _finish_auth 0 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:12:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 5 monclient: authenticate success, global_id 104436 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 monclient: _renew_subs 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc8ac140110 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 client.104436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 monclient: _renew_subs 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc8ac13db50 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 1 librados: init done 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc894001c60 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc894003a70 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 monclient: dump: 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:37.868 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a9ffb640 4 set_mon_vals no callback set 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fc894003c10 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc890005090 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(343..343 src has 1..343) ==== 5386+0+0 (unknown 2715816078 0 0) 0x7fc89404b7e0 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 client.104436.objecter ms_dispatch 0x7fc8ac000ba0 osd_map(343..343 src has 1..343) 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 3 client.104436.objecter handle_osd_map got epochs [343,343] > 0 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 3 client.104436.objecter handle_osd_map decoding full epoch 343 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 20 client.104436.objecter dump_active .. 0 homeless 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8ab7fe640 10 monclient: get_auth_request con 0x7fc88c03dfc0 auth_method 0 2024-09-16T03:12:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.852+0000 7fc8a3fff640 10 client.104436.objecter ms_handle_connect 0x7fc88c03dfc0 2024-09-16T03:12:37.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.860+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fc894016a60 con 0x7fc8ac0a4d20 2024-09-16T03:12:37.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.860+0000 7fc8a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:37.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:37.860+0000 7fc8a3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:38.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.092+0000 7fc8b1a6a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:12:38.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.092+0000 7fc8b1a6a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:38.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.092+0000 7fc8b1a6a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:38.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.092+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc890005380 con 0x7fc8ac0a4d20 2024-09-16T03:12:38.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8a3fff640 1 -- 172.21.15.45:0/543237788 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 1029790454) 0x7fc894016360 con 0x7fc8ac0a4d20 2024-09-16T03:12:38.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:38.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8a3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:38.100 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:12:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 10 librados: watch_flush enter 2024-09-16T03:12:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 10 librados: watch_flush exit 2024-09-16T03:12:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 20 client.104436.objecter shutdown clearing up homeless session... 2024-09-16T03:12:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 10 client.104436.objecter successfully canceled tick 2024-09-16T03:12:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 >> v1:172.21.15.46:6812/149360280 conn(0x7fc88c03dfc0 legacy=0x7fc88c040480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 10 monclient: shutdown 2024-09-16T03:12:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 >> v1:172.21.15.45:6789/0 conn(0x7fc8ac0a4d20 legacy=0x7fc8ac13d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 shutdown_connections 2024-09-16T03:12:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 >> 172.21.15.45:0/543237788 conn(0x7fc8ac0aa720 msgr2=0x7fc8ac1415d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 shutdown_connections 2024-09-16T03:12:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 1 -- 172.21.15.45:0/543237788 wait complete. 2024-09-16T03:12:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.096+0000 7fc8b1a6a640 1 librados: shutdown 2024-09-16T03:12:38.117 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:12:38.117 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:12:38.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:12:38.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient: get_monmap_and_config 2024-09-16T03:12:38.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient: build_initial_monmap 2024-09-16T03:12:38.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient: monmap: 2024-09-16T03:12:38.228 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:38.228 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.228 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:38.228547+0000 2024-09-16T03:12:38.228 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:38.228547+0000 2024-09-16T03:12:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:38.229 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 Processor -- start 2024-09-16T03:12:38.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 -- start start 2024-09-16T03:12:38.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient: init 2024-09-16T03:12:38.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:38.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:12:38.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient(hunting): picked mon.noname-a con 0x7f7cb4135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient(hunting): picked mon.noname-b con 0x7f7cb409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f7cb413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb409f7e0 con 0x7f7cb4135950 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb409b160 con 0x7f7cb413c010 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb409b440 con 0x7f7cb409fc40 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient(hunting): authenticate will time out at 1448.877792s 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb1d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f7cb409fc40 0x7f7cb413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40586/0 (socket says 172.21.15.45:40586) 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb1d9b640 1 -- 172.21.15.45:0/4226882278 learned_addr learned my addr 172.21.15.45:0/4226882278 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb259c640 1 --1- 172.21.15.45:0/4226882278 >> v1:172.21.15.45:6789/0 conn(0x7f7cb4135950 0x7f7cb4135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56950/0 (socket says 172.21.15.45:56950) 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3189792476 0 0) 0x7f7cb409f7e0 con 0x7f7cb4135950 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient(hunting): my global_id is 104439 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c9c003530 con 0x7f7cb4135950 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3559069059 0 0) 0x7f7cb409b440 con 0x7f7cb409fc40 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient(hunting): my global_id is 124100 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cb409f7e0 con 0x7f7cb409fc40 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1366410554 0 0) 0x7f7c9c003530 con 0x7f7cb4135950 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 >> v1:172.21.15.45:6790/0 conn(0x7f7cb409fc40 legacy=0x7f7cb413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 >> v1:172.21.15.46:6789/0 conn(0x7f7cb413c010 legacy=0x7f7cb413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: _finish_hunting 0 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 monclient: found mon.noname-a 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb4136440 con 0x7f7cb4135950 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: _finish_auth 0 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: _check_auth_tickets 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ca80016a0 con 0x7f7cb4135950 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:38.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: dump: 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 5 monclient: authenticate success, global_id 104439 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:38.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ca80018c0 con 0x7f7cb4135950 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 1 -- 172.21.15.45:0/4226882278 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ca8001d10 con 0x7f7cb4135950 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 4 set_mon_vals no callback set 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb159a640 10 monclient: dump: 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 10 monclient: shutdown 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/4226882278 >> v1:172.21.15.45:6789/0 conn(0x7f7cb4135950 legacy=0x7f7cb4135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/4226882278 shutdown_connections 2024-09-16T03:12:38.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/4226882278 >> 172.21.15.45:0/4226882278 conn(0x7f7cb40a2bd0 msgr2=0x7f7cb40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/4226882278 shutdown_connections 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.224+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/4226882278 wait complete. 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient: build_initial_monmap 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient: monmap: 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:38.244 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 librados: starting msgr at 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 librados: starting objecter 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 Processor -- start 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 -- start start 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 librados: setting wanted keys 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 librados: calling monclient init 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient: init 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient(hunting): picked mon.noname-a con 0x7f7cb409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f7cb4135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:38.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient(hunting): picked mon.noname-b con 0x7f7cb413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb4070340 con 0x7f7cb409fc40 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb4071b40 con 0x7f7cb413c010 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb40712a0 con 0x7f7cb4135950 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient(hunting): authenticate will time out at 1448.881102s 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb259c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7cb409fc40 0x7f7cb406e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56964/0 (socket says 172.21.15.45:56964) 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb1d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f7cb4135950 0x7f7cb406ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40598/0 (socket says 172.21.15.45:40598) 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb259c640 1 -- 172.21.15.45:0/2704602653 learned_addr learned my addr 172.21.15.45:0/2704602653 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 client.?.objecter ms_handle_connect 0x7f7cb409fc40 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 client.?.objecter ms_handle_connect 0x7f7cb4135950 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557335828 0 0) 0x7f7cb4070340 con 0x7f7cb409fc40 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): my global_id is 104442 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c94004ec0 con 0x7f7cb409fc40 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 174385069 0 0) 0x7f7cb40712a0 con 0x7f7cb4135950 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): my global_id is 124103 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cb4070340 con 0x7f7cb4135950 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 client.?.objecter ms_handle_connect 0x7f7cb413c010 2024-09-16T03:12:38.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2538393767 0 0) 0x7f7c94004ec0 con 0x7f7cb409fc40 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7cb40712a0 con 0x7f7cb409fc40 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1584896810 0 0) 0x7f7cb4070340 con 0x7f7cb4135950 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c94004ec0 con 0x7f7cb4135950 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ca8001cd0 con 0x7f7cb409fc40 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): dump: 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.248 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ca40018a0 con 0x7f7cb4135950 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): dump: 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.249 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667352797 0 0) 0x7f7cb4071b40 con 0x7f7cb413c010 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): my global_id is 22927 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cb4070340 con 0x7f7cb413c010 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1770233802 0 0) 0x7f7c94004ec0 con 0x7f7cb4135950 2024-09-16T03:12:38.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 >> v1:172.21.15.46:6789/0 conn(0x7f7cb413c010 legacy=0x7f7cb406f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 >> v1:172.21.15.45:6789/0 conn(0x7f7cb409fc40 legacy=0x7f7cb406e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 monclient: found mon.c 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb4070dc0 con 0x7f7cb4135950 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 5 monclient: authenticate success, global_id 124103 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient: _renew_subs 2024-09-16T03:12:38.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb409d5d0 con 0x7f7cb4135950 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 client.124103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient: _renew_subs 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7cb409db80 con 0x7f7cb4135950 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 librados: init done 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ca40047f0 con 0x7f7cb4135950 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ca4004b40 con 0x7f7cb4135950 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb0d99640 4 set_mon_vals no callback set 2024-09-16T03:12:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: dump: 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7c980065d0 con 0x7f7c98002870 2024-09-16T03:12:38.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 client.124103.objecter ms_handle_connect 0x7f7c98002870 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 client.124103.objecter resend_mon_ops 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 805312517 0 0) 0x7f7c980065d0 con 0x7f7c98002870 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient(hunting): my global_id is 104445 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c94004ec0 con 0x7f7c98002870 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7ca4010bb0 con 0x7f7cb4135950 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(343..343 src has 1..343) ==== 5386+0+0 (unknown 2715816078 0 0) 0x7f7ca404b5f0 con 0x7f7cb4135950 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 client.124103.objecter ms_dispatch 0x7f7cb4000ba0 osd_map(343..343 src has 1..343) 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 3 client.124103.objecter handle_osd_map got epochs [343,343] > 0 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 3 client.124103.objecter handle_osd_map decoding full epoch 343 2024-09-16T03:12:38.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 20 client.124103.objecter dump_active .. 0 homeless 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 771035251 0 0) 0x7f7c94004ec0 con 0x7f7c98002870 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c980065d0 con 0x7f7c98002870 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ca8004df0 con 0x7f7c98002870 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 10 monclient: dump: 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.255 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2422644904 0 0) 0x7f7c980065d0 con 0x7f7c98002870 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c9800a170 con 0x7f7c98002870 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.228+0000 7f7cb2d9d640 10 monclient: get_auth_request con 0x7f7c94042640 auth_method 0 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.232+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7ca8007480 con 0x7f7c98002870 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.232+0000 7f7ca2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.232+0000 7f7ca2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.232+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 >> v1:172.21.15.45:6789/0 conn(0x7f7c98002870 legacy=0x7f7c98004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.232+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 >> v1:172.21.15.45:6789/0 conn(0x7f7c98002870 legacy=0x7f7c98004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:38.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.232+0000 7f7ca2ffd640 10 client.124103.objecter ms_handle_connect 0x7f7c94042640 2024-09-16T03:12:38.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:12:38.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7c98009e60 con 0x7f7c98009120 2024-09-16T03:12:38.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 client.124103.objecter ms_handle_connect 0x7f7c98009120 2024-09-16T03:12:38.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 client.124103.objecter resend_mon_ops 2024-09-16T03:12:38.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 505859439 0 0) 0x7f7c98009e60 con 0x7f7c98009120 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 monclient(hunting): my global_id is 104448 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c980065d0 con 0x7f7c98009120 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1600111622 0 0) 0x7f7c980065d0 con 0x7f7c98009120 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c98009e60 con 0x7f7c98009120 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ca80055e0 con 0x7f7c98009120 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 monclient: dump: 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.259 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2550698556 0 0) 0x7f7c98009e60 con 0x7f7c98009120 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7c9800c400 con 0x7f7c98009120 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4137 (unknown 0 0 2714656532) 0x7f7ca80076b0 con 0x7f7c98009120 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:12:38.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 >> v1:172.21.15.45:6789/0 conn(0x7f7c98009120 legacy=0x7f7c980095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7ca2ffd640 1 -- 172.21.15.45:0/2704602653 >> v1:172.21.15.45:6789/0 conn(0x7f7c98009120 legacy=0x7f7c980095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:38.261 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:12:38.261 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:38.261 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:38.261 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:12:38.261 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 32, 2024-09-16T03:12:38.261 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 217337, 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:38.262 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:38.263 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:12:38.264 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:38.265 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:12:38.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.268 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:38.269 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:38.270 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.271 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.272 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.273 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:12:38.273 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:12:38.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 10 librados: watch_flush enter 2024-09-16T03:12:38.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 10 librados: watch_flush exit 2024-09-16T03:12:38.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 20 client.124103.objecter shutdown clearing up homeless session... 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 10 client.124103.objecter successfully canceled tick 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 >> v1:172.21.15.46:6812/149360280 conn(0x7f7c94042640 legacy=0x7f7c94044b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 10 monclient: shutdown 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 >> v1:172.21.15.45:6790/0 conn(0x7f7cb4135950 legacy=0x7f7cb406ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb2d9d640 1 -- 172.21.15.45:0/2704602653 reap_dead start 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb2d9d640 1 -- 172.21.15.45:0/2704602653 reap_dead start 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 shutdown_connections 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 >> 172.21.15.45:0/2704602653 conn(0x7f7cb40a2bd0 msgr2=0x7f7cb413f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 shutdown_connections 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 1 -- 172.21.15.45:0/2704602653 wait complete. 2024-09-16T03:12:38.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.236+0000 7f7cb8a8f640 1 librados: shutdown 2024-09-16T03:12:38.276 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:12:38.276 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:12:38.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:12:38.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient: get_monmap_and_config 2024-09-16T03:12:38.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient: build_initial_monmap 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient: monmap: 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:38.384603+0000 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:38.384603+0000 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 Processor -- start 2024-09-16T03:12:38.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 -- start start 2024-09-16T03:12:38.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient: init 2024-09-16T03:12:38.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:38.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:38.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient(hunting): picked mon.noname-c con 0x7fda30135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient(hunting): picked mon.noname-a con 0x7fda3009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient(hunting): picked mon.noname-b con 0x7fda3013c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3009f7e0 con 0x7fda3009fc40 2024-09-16T03:12:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3009b160 con 0x7fda30135950 2024-09-16T03:12:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3009b440 con 0x7fda3013c010 2024-09-16T03:12:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient(hunting): authenticate will time out at 1449.033885s 2024-09-16T03:12:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2f7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fda3013c010 0x7fda3013e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40604/0 (socket says 172.21.15.45:40604) 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2e7fc640 1 --1- 172.21.15.45:0/2658675232 >> v1:172.21.15.45:6789/0 conn(0x7fda3009fc40 0x7fda3013a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57004/0 (socket says 172.21.15.45:57004) 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2f7fe640 1 -- 172.21.15.45:0/2658675232 learned_addr learned my addr 172.21.15.45:0/2658675232 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1558761699 0 0) 0x7fda3009f7e0 con 0x7fda3009fc40 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient(hunting): my global_id is 104451 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda18003530 con 0x7fda3009fc40 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4231697087 0 0) 0x7fda3009b440 con 0x7fda3013c010 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient(hunting): my global_id is 124106 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda3009f7e0 con 0x7fda3013c010 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2404565078 0 0) 0x7fda18003530 con 0x7fda3009fc40 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 >> v1:172.21.15.45:6790/0 conn(0x7fda3013c010 legacy=0x7fda3013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 >> v1:172.21.15.46:6789/0 conn(0x7fda30135950 legacy=0x7fda30135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 monclient: found mon.noname-a 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda30136440 con 0x7fda3009fc40 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: _finish_auth 0 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda200015d0 con 0x7fda3009fc40 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 5 monclient: authenticate success, global_id 104451 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: dump: 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda200017f0 con 0x7fda3009fc40 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 4 set_mon_vals no callback set 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 1 -- 172.21.15.45:0/2658675232 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda20001c40 con 0x7fda3009fc40 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda2dffb640 10 monclient: dump: 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 10 monclient: shutdown 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 -- 172.21.15.45:0/2658675232 >> v1:172.21.15.45:6789/0 conn(0x7fda3009fc40 legacy=0x7fda3013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 -- 172.21.15.45:0/2658675232 shutdown_connections 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 -- 172.21.15.45:0/2658675232 >> 172.21.15.45:0/2658675232 conn(0x7fda300a2bd0 msgr2=0x7fda300a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 -- 172.21.15.45:0/2658675232 shutdown_connections 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.380+0000 7fda353a0640 1 -- 172.21.15.45:0/2658675232 wait complete. 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient: build_initial_monmap 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient: monmap: 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 librados: starting msgr at 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 librados: starting objecter 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 Processor -- start 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 -- start start 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 librados: setting wanted keys 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 librados: calling monclient init 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient: init 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient(hunting): picked mon.noname-a con 0x7fda3009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient(hunting): picked mon.noname-c con 0x7fda30135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient(hunting): picked mon.noname-b con 0x7fda3013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3013bb00 con 0x7fda3009fc40 2024-09-16T03:12:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda30071cf0 con 0x7fda3013c010 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3013b6c0 con 0x7fda30135950 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient(hunting): authenticate will time out at 1449.036902s 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda2effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fda3009fc40 0x7fda3006e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57018/0 (socket says 172.21.15.45:57018) 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda2effd640 1 -- 172.21.15.45:0/2095205678 learned_addr learned my addr 172.21.15.45:0/2095205678 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.?.objecter ms_handle_connect 0x7fda3009fc40 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.?.objecter ms_handle_connect 0x7fda30135950 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3537863958 0 0) 0x7fda3013bb00 con 0x7fda3009fc40 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): my global_id is 104454 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda1c004ec0 con 0x7fda3009fc40 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266587655 0 0) 0x7fda3013b6c0 con 0x7fda30135950 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): my global_id is 124109 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda3013bb00 con 0x7fda30135950 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.?.objecter ms_handle_connect 0x7fda3013c010 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 319487048 0 0) 0x7fda3013bb00 con 0x7fda30135950 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda3013b6c0 con 0x7fda30135950 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3873364537 0 0) 0x7fda1c004ec0 con 0x7fda3009fc40 2024-09-16T03:12:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda3013bb00 con 0x7fda3009fc40 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda20001b40 con 0x7fda30135950 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): dump: 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.403 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda240019c0 con 0x7fda3009fc40 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): dump: 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 322302681 0 0) 0x7fda3013b6c0 con 0x7fda30135950 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 >> v1:172.21.15.46:6789/0 conn(0x7fda3013c010 legacy=0x7fda301d2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 >> v1:172.21.15.45:6789/0 conn(0x7fda3009fc40 legacy=0x7fda3006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: _finish_hunting 0 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 monclient: found mon.c 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda30071450 con 0x7fda30135950 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: _finish_auth 0 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: _check_auth_tickets 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 5 monclient: authenticate success, global_id 124109 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient: _renew_subs 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda30071150 con 0x7fda30135950 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 client.124109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:38.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient: _renew_subs 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fda3006f600 con 0x7fda30135950 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 librados: init done 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda20001750 con 0x7fda30135950 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda20004b40 con 0x7fda30135950 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda2d7fa640 4 set_mon_vals no callback set 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: dump: 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fda100065d0 con 0x7fda10002870 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.124109.objecter ms_handle_connect 0x7fda10002870 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.124109.objecter resend_mon_ops 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fda20010bb0 con 0x7fda30135950 2024-09-16T03:12:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(343..343 src has 1..343) ==== 5386+0+0 (unknown 2715816078 0 0) 0x7fda2004b6a0 con 0x7fda30135950 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.124109.objecter ms_dispatch 0x7fda30000ba0 osd_map(343..343 src has 1..343) 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 3 client.124109.objecter handle_osd_map got epochs [343,343] > 0 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 3 client.124109.objecter handle_osd_map decoding full epoch 343 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 20 client.124109.objecter dump_active .. 0 homeless 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270815089 0 0) 0x7fda100065d0 con 0x7fda10002870 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient(hunting): my global_id is 124112 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda3013b6c0 con 0x7fda10002870 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4133415460 0 0) 0x7fda3013b6c0 con 0x7fda10002870 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda100065d0 con 0x7fda10002870 2024-09-16T03:12:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda24003a70 con 0x7fda10002870 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda2f7fe640 10 monclient: get_auth_request con 0x7fda1c03e0b0 auth_method 0 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: dump: 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.411 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2407891651 0 0) 0x7fda100065d0 con 0x7fda10002870 2024-09-16T03:12:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda1000a170 con 0x7fda10002870 2024-09-16T03:12:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 client.124109.objecter ms_handle_connect 0x7fda1c03e0b0 2024-09-16T03:12:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fda240057d0 con 0x7fda10002870 2024-09-16T03:12:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 >> v1:172.21.15.45:6790/0 conn(0x7fda10002870 legacy=0x7fda10004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.384+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 >> v1:172.21.15.45:6790/0 conn(0x7fda10002870 legacy=0x7fda10004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fda10009e60 con 0x7fda10009120 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 client.124109.objecter ms_handle_connect 0x7fda10009120 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 client.124109.objecter resend_mon_ops 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1659314320 0 0) 0x7fda10009e60 con 0x7fda10009120 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 monclient(hunting): my global_id is 124115 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda100065d0 con 0x7fda10009120 2024-09-16T03:12:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2729264616 0 0) 0x7fda100065d0 con 0x7fda10009120 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda10009e60 con 0x7fda10009120 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda24004160 con 0x7fda10009120 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 monclient: dump: 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 913648221 0 0) 0x7fda10009e60 con 0x7fda10009120 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda1000c400 con 0x7fda10009120 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3965 (unknown 0 0 1886227053) 0x7fda24003f80 con 0x7fda10009120 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 >> v1:172.21.15.45:6790/0 conn(0x7fda10009120 legacy=0x7fda100095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda177fe640 1 -- 172.21.15.45:0/2095205678 >> v1:172.21.15.45:6790/0 conn(0x7fda10009120 legacy=0x7fda100095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:38.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 32, 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 623, 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:12:38.416 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:12:38.417 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.418 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:12:38.419 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:38.420 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:12:38.421 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:38.422 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:38.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.424 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:38.425 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:12:38.426 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:12:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 10 librados: watch_flush enter 2024-09-16T03:12:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 10 librados: watch_flush exit 2024-09-16T03:12:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 20 client.124109.objecter shutdown clearing up homeless session... 2024-09-16T03:12:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 10 client.124109.objecter successfully canceled tick 2024-09-16T03:12:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 >> v1:172.21.15.46:6812/149360280 conn(0x7fda1c03e0b0 legacy=0x7fda1c040570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 10 monclient: shutdown 2024-09-16T03:12:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 >> v1:172.21.15.45:6790/0 conn(0x7fda30135950 legacy=0x7fda3006ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda2f7fe640 1 -- 172.21.15.45:0/2095205678 reap_dead start 2024-09-16T03:12:38.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda2f7fe640 1 -- 172.21.15.45:0/2095205678 reap_dead start 2024-09-16T03:12:38.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 shutdown_connections 2024-09-16T03:12:38.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 >> 172.21.15.45:0/2095205678 conn(0x7fda300a2bd0 msgr2=0x7fda3013ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:38.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 shutdown_connections 2024-09-16T03:12:38.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 1 -- 172.21.15.45:0/2095205678 wait complete. 2024-09-16T03:12:38.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.392+0000 7fda353a0640 1 librados: shutdown 2024-09-16T03:12:38.429 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:12:38.429 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:12:38.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:12:38.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.532+0000 7f7ef2312640 10 monclient: get_monmap_and_config 2024-09-16T03:12:38.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.532+0000 7f7ef2312640 10 monclient: build_initial_monmap 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.532+0000 7f7ef2312640 10 monclient: monmap: 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:38.539939+0000 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:38.539939+0000 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:38.540 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 Processor -- start 2024-09-16T03:12:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 -- start start 2024-09-16T03:12:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient: init 2024-09-16T03:12:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient(hunting): picked mon.noname-c con 0x7f7eec135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient(hunting): picked mon.noname-b con 0x7f7eec09fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient(hunting): picked mon.noname-a con 0x7f7eec13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec09f7e0 con 0x7f7eec13c010 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec09b160 con 0x7f7eec135950 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec09b440 con 0x7f7eec09fc40 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient(hunting): authenticate will time out at 1449.189527s 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeb7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f7eec09fc40 0x7f7eec13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40626/0 (socket says 172.21.15.45:40626) 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeb7fe640 1 -- 172.21.15.45:0/3147409058 learned_addr learned my addr 172.21.15.45:0/3147409058 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701603669 0 0) 0x7f7eec09b440 con 0x7f7eec09fc40 2024-09-16T03:12:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient(hunting): my global_id is 124118 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ed4003530 con 0x7f7eec09fc40 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308360063 0 0) 0x7f7eec09f7e0 con 0x7f7eec13c010 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient(hunting): my global_id is 104457 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eec09b440 con 0x7f7eec13c010 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3348313208 0 0) 0x7f7ed4003530 con 0x7f7eec09fc40 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 >> v1:172.21.15.46:6789/0 conn(0x7f7eec135950 legacy=0x7f7eec135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 >> v1:172.21.15.45:6789/0 conn(0x7f7eec13c010 legacy=0x7f7eec13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: _finish_hunting 0 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 monclient: found mon.noname-b 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eec136440 con 0x7f7eec09fc40 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: _finish_auth 0 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: _check_auth_tickets 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ee00016a0 con 0x7f7eec09fc40 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: dump: 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 5 monclient: authenticate success, global_id 124118 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ee00018c0 con 0x7f7eec09fc40 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 1 -- 172.21.15.45:0/3147409058 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ee0001d10 con 0x7f7eec09fc40 2024-09-16T03:12:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7eeaffd640 10 monclient: dump: 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:38.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 4 set_mon_vals no callback set 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient: shutdown 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 -- 172.21.15.45:0/3147409058 >> v1:172.21.15.45:6790/0 conn(0x7f7eec09fc40 legacy=0x7f7eec13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 -- 172.21.15.45:0/3147409058 shutdown_connections 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 -- 172.21.15.45:0/3147409058 >> 172.21.15.45:0/3147409058 conn(0x7f7eec0a2bd0 msgr2=0x7f7eec0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 -- 172.21.15.45:0/3147409058 shutdown_connections 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 -- 172.21.15.45:0/3147409058 wait complete. 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient: build_initial_monmap 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 10 monclient: monmap: 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 librados: starting msgr at 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.536+0000 7f7ef2312640 1 librados: starting objecter 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 Processor -- start 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 -- start start 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 librados: setting wanted keys 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 librados: calling monclient init 2024-09-16T03:12:38.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient: init 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient(hunting): picked mon.noname-b con 0x7f7eec09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient(hunting): picked mon.noname-c con 0x7f7eec135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient(hunting): picked mon.noname-a con 0x7f7eec13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec0714f0 con 0x7f7eec13c010 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec0717d0 con 0x7f7eec09fc40 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec071d90 con 0x7f7eec135950 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient(hunting): authenticate will time out at 1449.192670s 2024-09-16T03:12:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7eeb7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f7eec135950 0x7f7eec1cfde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40630/0 (socket says 172.21.15.45:40630) 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7eeb7fe640 1 -- 172.21.15.45:0/2850214888 learned_addr learned my addr 172.21.15.45:0/2850214888 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef08af640 1 --1- 172.21.15.45:0/2850214888 >> v1:172.21.15.45:6789/0 conn(0x7f7eec13c010 0x7f7eec1d3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57036/0 (socket says 172.21.15.45:57036) 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.?.objecter ms_handle_connect 0x7f7eec135950 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.?.objecter ms_handle_connect 0x7f7eec13c010 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 580163560 0 0) 0x7f7eec071d90 con 0x7f7eec135950 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): my global_id is 124121 2024-09-16T03:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ed8004ec0 con 0x7f7eec135950 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1050137901 0 0) 0x7f7eec0714f0 con 0x7f7eec13c010 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): my global_id is 104460 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eec071d90 con 0x7f7eec13c010 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.?.objecter ms_handle_connect 0x7f7eec09fc40 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2289835813 0 0) 0x7f7ed8004ec0 con 0x7f7eec135950 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7eec0714f0 con 0x7f7eec135950 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ee0001cd0 con 0x7f7eec135950 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): dump: 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1876047227 0 0) 0x7f7eec071d90 con 0x7f7eec13c010 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ed8004ec0 con 0x7f7eec13c010 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ee4001870 con 0x7f7eec13c010 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): dump: 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1486905806 0 0) 0x7f7eec0717d0 con 0x7f7eec09fc40 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): my global_id is 22939 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eec071d90 con 0x7f7eec09fc40 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 692244242 0 0) 0x7f7eec0714f0 con 0x7f7eec135950 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 >> v1:172.21.15.46:6789/0 conn(0x7f7eec09fc40 legacy=0x7f7eec1cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 >> v1:172.21.15.45:6789/0 conn(0x7f7eec13c010 legacy=0x7f7eec1d3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:12:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 monclient: found mon.c 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eec071ab0 con 0x7f7eec135950 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: _finish_auth 0 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 5 monclient: authenticate success, global_id 124121 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient: _renew_subs 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7eec070110 con 0x7f7eec135950 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 client.124121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient: _renew_subs 2024-09-16T03:12:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7eec06f590 con 0x7f7eec135950 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 librados: init done 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ee0001840 con 0x7f7eec135950 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7ee0004c40 con 0x7f7eec135950 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7eea7fc640 4 set_mon_vals no callback set 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: dump: 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7ecc0065d0 con 0x7f7ecc002870 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7ee0004de0 con 0x7f7eec135950 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(343..343 src has 1..343) ==== 5386+0+0 (unknown 2715816078 0 0) 0x7f7ee004b7a0 con 0x7f7eec135950 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.124121.objecter ms_dispatch 0x7f7eec000ba0 osd_map(343..343 src has 1..343) 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 3 client.124121.objecter handle_osd_map got epochs [343,343] > 0 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 3 client.124121.objecter handle_osd_map decoding full epoch 343 2024-09-16T03:12:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 20 client.124121.objecter dump_active .. 0 homeless 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.124121.objecter ms_handle_connect 0x7f7ecc002870 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.124121.objecter resend_mon_ops 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ef08af640 10 monclient: get_auth_request con 0x7f7ed803e300 auth_method 0 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1186057172 0 0) 0x7f7ecc0065d0 con 0x7f7ecc002870 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient(hunting): my global_id is 22942 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eec0714f0 con 0x7f7ecc002870 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 client.124121.objecter ms_handle_connect 0x7f7ed803e300 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3789063560 0 0) 0x7f7eec0714f0 con 0x7f7ecc002870 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ecc0065d0 con 0x7f7ecc002870 2024-09-16T03:12:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7edc001a90 con 0x7f7ecc002870 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 10 monclient: dump: 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4050948074 0 0) 0x7f7ecc0065d0 con 0x7f7ecc002870 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.540+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ecc00a170 con 0x7f7ecc002870 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.544+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7edc001db0 con 0x7f7ecc002870 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.544+0000 7f7ee8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.544+0000 7f7ee8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.544+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 >> v1:172.21.15.46:6789/0 conn(0x7f7ecc002870 legacy=0x7f7ecc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.544+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 >> v1:172.21.15.46:6789/0 conn(0x7f7ecc002870 legacy=0x7f7ecc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ef2312640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7ecc009e80 con 0x7f7ecc009140 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ee8ff9640 10 client.124121.objecter ms_handle_connect 0x7f7ecc009140 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ee8ff9640 10 client.124121.objecter resend_mon_ops 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 284860023 0 0) 0x7f7ecc009e80 con 0x7f7ecc009140 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ee8ff9640 10 monclient(hunting): my global_id is 22945 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.548+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ecc0065d0 con 0x7f7ecc009140 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3565374391 0 0) 0x7f7ecc0065d0 con 0x7f7ecc009140 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ecc009e80 con 0x7f7ecc009140 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7edc003b20 con 0x7f7ecc009140 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 10 monclient: dump: 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.569 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1491306639 0 0) 0x7f7ecc009e80 con 0x7f7ecc009140 2024-09-16T03:12:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ecc00c420 con 0x7f7ecc009140 2024-09-16T03:12:38.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 1557963890) 0x7f7edc003ab0 con 0x7f7ecc009140 2024-09-16T03:12:38.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:12:38.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:38.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 >> v1:172.21.15.46:6789/0 conn(0x7f7ecc009140 legacy=0x7f7ecc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/2850214888 >> v1:172.21.15.46:6789/0 conn(0x7f7ecc009140 legacy=0x7f7ecc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 32, 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 480358, 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:12:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:38.573 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:12:38.574 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:38.575 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:38.576 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:38.577 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:38.578 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.581 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:38.582 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:12:38.583 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:12:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 10 librados: watch_flush enter 2024-09-16T03:12:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 10 librados: watch_flush exit 2024-09-16T03:12:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 20 client.124121.objecter shutdown clearing up homeless session... 2024-09-16T03:12:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 10 client.124121.objecter successfully canceled tick 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 >> v1:172.21.15.46:6812/149360280 conn(0x7f7ed803e300 legacy=0x7f7ed80407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 10 monclient: shutdown 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 >> v1:172.21.15.45:6790/0 conn(0x7f7eec135950 legacy=0x7f7eec1cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef08af640 1 -- 172.21.15.45:0/2850214888 reap_dead start 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef08af640 1 -- 172.21.15.45:0/2850214888 reap_dead start 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 shutdown_connections 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 >> 172.21.15.45:0/2850214888 conn(0x7f7eec0a2bd0 msgr2=0x7f7eec0a0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 shutdown_connections 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 1 -- 172.21.15.45:0/2850214888 wait complete. 2024-09-16T03:12:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.552+0000 7f7ef2312640 1 librados: shutdown 2024-09-16T03:12:38.586 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:12:38.586 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:12:38.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:12:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 10 monclient: get_monmap_and_config 2024-09-16T03:12:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 10 monclient: build_initial_monmap 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 10 monclient: monmap: 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:38.694857+0000 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:38.694857+0000 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:38.695 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 1 Processor -- start 2024-09-16T03:12:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 1 -- start start 2024-09-16T03:12:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 10 monclient: init 2024-09-16T03:12:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:12:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 10 monclient(hunting): picked mon.noname-a con 0x7ff67c135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 10 monclient(hunting): picked mon.noname-c con 0x7ff67c09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 10 monclient(hunting): picked mon.noname-b con 0x7ff67c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.688+0000 7ff680d46640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff67c09f7e0 con 0x7ff67c135950 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff67c09b160 con 0x7ff67c09fc40 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff67c09b440 con 0x7ff67c13c010 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient(hunting): authenticate will time out at 1449.344046s 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67a59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff67c135950 0x7ff67c135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57040/0 (socket says 172.21.15.45:57040) 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67a59c640 1 -- 172.21.15.45:0/1304845133 learned_addr learned my addr 172.21.15.45:0/1304845133 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67ad9d640 1 --1- 172.21.15.45:0/1304845133 >> v1:172.21.15.45:6790/0 conn(0x7ff67c13c010 0x7ff67c13e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40640/0 (socket says 172.21.15.45:40640) 2024-09-16T03:12:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 555541151 0 0) 0x7ff67c09f7e0 con 0x7ff67c135950 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient(hunting): my global_id is 104463 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff664003530 con 0x7ff67c135950 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176386550 0 0) 0x7ff67c09b440 con 0x7ff67c13c010 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient(hunting): my global_id is 124124 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff67c09f7e0 con 0x7ff67c13c010 2024-09-16T03:12:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4057684048 0 0) 0x7ff67c09f7e0 con 0x7ff67c13c010 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 >> v1:172.21.15.46:6789/0 conn(0x7ff67c09fc40 legacy=0x7ff67c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 >> v1:172.21.15.45:6789/0 conn(0x7ff67c135950 legacy=0x7ff67c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: _finish_hunting 0 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 monclient: found mon.noname-b 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67c136440 con 0x7ff67c13c010 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: _finish_auth 0 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: _check_auth_tickets 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff674001680 con 0x7ff67c13c010 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 5 monclient: authenticate success, global_id 124124 2024-09-16T03:12:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: dump: 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.707 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff674001820 con 0x7ff67c13c010 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 1 -- 172.21.15.45:0/1304845133 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff674001c70 con 0x7ff67c13c010 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff67959a640 10 monclient: dump: 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 4 set_mon_vals no callback set 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: shutdown 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- 172.21.15.45:0/1304845133 >> v1:172.21.15.45:6790/0 conn(0x7ff67c13c010 legacy=0x7ff67c13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- 172.21.15.45:0/1304845133 shutdown_connections 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- 172.21.15.45:0/1304845133 >> 172.21.15.45:0/1304845133 conn(0x7ff67c0a2bd0 msgr2=0x7ff67c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- 172.21.15.45:0/1304845133 shutdown_connections 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- 172.21.15.45:0/1304845133 wait complete. 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: build_initial_monmap 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: monmap: 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 librados: starting msgr at 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 librados: starting objecter 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 Processor -- start 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- start start 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 librados: setting wanted keys 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 librados: calling monclient init 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: init 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient(hunting): picked mon.noname-b con 0x7ff67c09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient(hunting): picked mon.noname-a con 0x7ff67c135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:38.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient(hunting): picked mon.noname-c con 0x7ff67c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff67c13b6f0 con 0x7ff67c135950 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff67c09d2c0 con 0x7ff67c09fc40 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff67c09ea90 con 0x7ff67c13c010 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient(hunting): authenticate will time out at 1449.346810s 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff679d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff67c135950 0x7ff67c1d1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57046/0 (socket says 172.21.15.45:57046) 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff679d9b640 1 -- 172.21.15.45:0/1291612748 learned_addr learned my addr 172.21.15.45:0/1291612748 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 client.?.objecter ms_handle_connect 0x7ff67c135950 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 client.?.objecter ms_handle_connect 0x7ff67c13c010 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3825278340 0 0) 0x7ff67c09b160 con 0x7ff67c135950 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): my global_id is 104466 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff65c004ec0 con 0x7ff67c135950 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2229991097 0 0) 0x7ff67c09ea90 con 0x7ff67c13c010 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): my global_id is 124127 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff67c09b160 con 0x7ff67c13c010 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 client.?.objecter ms_handle_connect 0x7ff67c09fc40 2024-09-16T03:12:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2941417582 0 0) 0x7ff65c004ec0 con 0x7ff67c135950 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff67c09ea90 con 0x7ff67c135950 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff66c001d20 con 0x7ff67c135950 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): dump: 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.714 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3360333629 0 0) 0x7ff67c09b160 con 0x7ff67c13c010 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff65c004ec0 con 0x7ff67c13c010 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff674001b70 con 0x7ff67c13c010 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): dump: 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.715 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911812468 0 0) 0x7ff670000f70 con 0x7ff67c09fc40 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient(hunting): my global_id is 22951 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff67c09b160 con 0x7ff67c09fc40 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1366005001 0 0) 0x7ff67c09ea90 con 0x7ff67c135950 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 >> v1:172.21.15.45:6790/0 conn(0x7ff67c13c010 legacy=0x7ff67c1d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 >> v1:172.21.15.46:6789/0 conn(0x7ff67c09fc40 legacy=0x7ff67c1ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient: _finish_hunting 0 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 monclient: found mon.a 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:38.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67c09ed70 con 0x7ff67c135950 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient: _finish_auth 0 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient: _check_auth_tickets 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 5 monclient: authenticate success, global_id 104466 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: _renew_subs 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff67c074310 con 0x7ff67c135950 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 client.104466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: _renew_subs 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff67c0737d0 con 0x7ff67c135950 2024-09-16T03:12:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff680d46640 1 librados: init done 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff66c0039a0 con 0x7ff67c135950 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.692+0000 7ff66affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff66c003db0 con 0x7ff67c135950 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff678d99640 4 set_mon_vals no callback set 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 10 monclient: dump: 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:38.719 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7ff66c003fd0 con 0x7ff67c135950 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff680d46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff680d46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff680d46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff660005090 con 0x7ff67c135950 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(343..343 src has 1..343) ==== 5386+0+0 (unknown 2715816078 0 0) 0x7ff66c04bc40 con 0x7ff67c135950 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 10 client.104466.objecter ms_dispatch 0x7ff67c000ba0 osd_map(343..343 src has 1..343) 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 3 client.104466.objecter handle_osd_map got epochs [343,343] > 0 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 3 client.104466.objecter handle_osd_map decoding full epoch 343 2024-09-16T03:12:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 20 client.104466.objecter dump_active .. 0 homeless 2024-09-16T03:12:38.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff67a59c640 10 monclient: get_auth_request con 0x7ff65c03e1f0 auth_method 0 2024-09-16T03:12:38.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.696+0000 7ff66affd640 10 client.104466.objecter ms_handle_connect 0x7ff65c03e1f0 2024-09-16T03:12:38.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.700+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7ff67c13b6f0 con 0x7ff67c135950 2024-09-16T03:12:38.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.700+0000 7ff66affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:38.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.700+0000 7ff66affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:38.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.928+0000 7ff680d46640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:12:38.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.928+0000 7ff680d46640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:12:38.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.928+0000 7ff680d46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:38.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.928+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff660005c50 con 0x7ff67c135950 2024-09-16T03:12:38.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.928+0000 7ff66affd640 1 -- 172.21.15.45:0/1291612748 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7ff66c016f40 con 0x7ff67c135950 2024-09-16T03:12:38.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.928+0000 7ff66affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:12:38.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.928+0000 7ff66affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:38.935 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.928+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.936 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.928+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=10,kv=13,logm=56} crc {auth=480680558,config=1097430493,health=1308591922,kv=505194352,logm=4121360783}) 2024-09-16T03:12:38.936 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.928+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=12,kv=13,logm=55} crc {auth=2664181157,config=1097430493,health=1995790572,kv=3981266340,logm=401851616}) 2024-09-16T03:12:38.936 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.928+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.936 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.928+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=10,kv=13,logm=56} crc {auth=480680558,config=1097430493,health=1308591922,kv=505194352,logm=4121360783}) 2024-09-16T03:12:38.936 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.928+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=11,kv=13,logm=56} crc {auth=4075155004,config=1097430493,health=1410898352,kv=4154458582,logm=2950084002}) 2024-09-16T03:12:38.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 10 librados: watch_flush enter 2024-09-16T03:12:38.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 10 librados: watch_flush exit 2024-09-16T03:12:38.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 20 client.104466.objecter shutdown clearing up homeless session... 2024-09-16T03:12:38.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 10 client.104466.objecter successfully canceled tick 2024-09-16T03:12:38.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 >> v1:172.21.15.46:6812/149360280 conn(0x7ff65c03e1f0 legacy=0x7ff65c0406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 10 monclient: shutdown 2024-09-16T03:12:38.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:38.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 >> v1:172.21.15.45:6789/0 conn(0x7ff67c135950 legacy=0x7ff67c1d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:38.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 shutdown_connections 2024-09-16T03:12:38.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 >> 172.21.15.45:0/1291612748 conn(0x7ff67c0a2bd0 msgr2=0x7ff67c13ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:38.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 shutdown_connections 2024-09-16T03:12:38.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 1 -- 172.21.15.45:0/1291612748 wait complete. 2024-09-16T03:12:38.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:38.932+0000 7ff680d46640 1 librados: shutdown 2024-09-16T03:12:38.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.932+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.932+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590657085}) 2024-09-16T03:12:38.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.932+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4187415223}) 2024-09-16T03:12:38.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.932+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.932+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590657085}) 2024-09-16T03:12:38.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.932+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3014192190}) 2024-09-16T03:12:38.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.936+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.936+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893408453}) 2024-09-16T03:12:38.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.936+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=323557517}) 2024-09-16T03:12:38.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.936+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.936+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893408453}) 2024-09-16T03:12:38.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.936+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905991824}) 2024-09-16T03:12:38.943 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.936+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.944 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786625615}) 2024-09-16T03:12:38.944 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2447571224}) 2024-09-16T03:12:38.944 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.944 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786625615}) 2024-09-16T03:12:38.944 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4251069726}) 2024-09-16T03:12:38.946 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.946 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888792392}) 2024-09-16T03:12:38.946 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143357053}) 2024-09-16T03:12:38.946 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.946 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888792392}) 2024-09-16T03:12:38.946 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.940+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2523947417}) 2024-09-16T03:12:38.948 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.948 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1938698590}) 2024-09-16T03:12:38.948 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133178455}) 2024-09-16T03:12:38.948 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.948 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1938698590}) 2024-09-16T03:12:38.949 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74263834}) 2024-09-16T03:12:38.950 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.950 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856263085}) 2024-09-16T03:12:38.950 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4233236525}) 2024-09-16T03:12:38.950 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.950 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856263085}) 2024-09-16T03:12:38.950 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.944+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008933546}) 2024-09-16T03:12:38.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668458648}) 2024-09-16T03:12:38.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1364424110}) 2024-09-16T03:12:38.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.953 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668458648}) 2024-09-16T03:12:38.953 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2693398182}) 2024-09-16T03:12:38.954 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.955 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=429412669}) 2024-09-16T03:12:38.955 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2536779287}) 2024-09-16T03:12:38.955 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.955 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=429412669}) 2024-09-16T03:12:38.955 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.948+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2441156082}) 2024-09-16T03:12:38.956 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:12:38.957 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.957 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853952537}) 2024-09-16T03:12:38.957 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456568219}) 2024-09-16T03:12:38.957 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.957 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853952537}) 2024-09-16T03:12:38.957 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1729606254}) 2024-09-16T03:12:38.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976851504}) 2024-09-16T03:12:38.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101820855}) 2024-09-16T03:12:38.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976851504}) 2024-09-16T03:12:38.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3997127937}) 2024-09-16T03:12:38.959 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306857680}) 2024-09-16T03:12:38.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4099390638}) 2024-09-16T03:12:38.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306857680}) 2024-09-16T03:12:38.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.952+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235657427}) 2024-09-16T03:12:38.961 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.961 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1200940879}) 2024-09-16T03:12:38.961 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4102719165}) 2024-09-16T03:12:38.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1200940879}) 2024-09-16T03:12:38.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543749107}) 2024-09-16T03:12:38.963 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.963 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1930823399}) 2024-09-16T03:12:38.963 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2890496531}) 2024-09-16T03:12:38.963 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.963 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1930823399}) 2024-09-16T03:12:38.963 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.956+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3846746736}) 2024-09-16T03:12:38.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192134333}) 2024-09-16T03:12:38.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=497886916}) 2024-09-16T03:12:38.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192134333}) 2024-09-16T03:12:38.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739831213}) 2024-09-16T03:12:38.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197737380}) 2024-09-16T03:12:38.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828424345}) 2024-09-16T03:12:38.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197737380}) 2024-09-16T03:12:38.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.960+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032591184}) 2024-09-16T03:12:38.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748835085}) 2024-09-16T03:12:38.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3722000466}) 2024-09-16T03:12:38.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748835085}) 2024-09-16T03:12:38.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376309885}) 2024-09-16T03:12:38.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2210379927}) 2024-09-16T03:12:38.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=986827161}) 2024-09-16T03:12:38.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2210379927}) 2024-09-16T03:12:38.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958386533}) 2024-09-16T03:12:38.971 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.971 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452046691}) 2024-09-16T03:12:38.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1600499567}) 2024-09-16T03:12:38.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452046691}) 2024-09-16T03:12:38.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.964+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2893593048}) 2024-09-16T03:12:38.973 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.973 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648755942}) 2024-09-16T03:12:38.973 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934118576}) 2024-09-16T03:12:38.974 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.974 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648755942}) 2024-09-16T03:12:38.974 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4046818037}) 2024-09-16T03:12:38.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1655948443}) 2024-09-16T03:12:38.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=569335452}) 2024-09-16T03:12:38.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1655948443}) 2024-09-16T03:12:38.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.968+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740724473}) 2024-09-16T03:12:38.976 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.976 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298222305}) 2024-09-16T03:12:38.976 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747894602}) 2024-09-16T03:12:38.977 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.977 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298222305}) 2024-09-16T03:12:38.977 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1117692595}) 2024-09-16T03:12:38.978 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.978 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048171265}) 2024-09-16T03:12:38.978 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2659546848}) 2024-09-16T03:12:38.978 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.978 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048171265}) 2024-09-16T03:12:38.978 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.972+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3254986238}) 2024-09-16T03:12:38.980 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.980 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1322153722}) 2024-09-16T03:12:38.980 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783987863}) 2024-09-16T03:12:38.980 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1322153722}) 2024-09-16T03:12:38.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=667751819}) 2024-09-16T03:12:38.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730378828}) 2024-09-16T03:12:38.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978801512}) 2024-09-16T03:12:38.982 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.982 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730378828}) 2024-09-16T03:12:38.982 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1898966169}) 2024-09-16T03:12:38.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426618516}) 2024-09-16T03:12:38.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1542565663}) 2024-09-16T03:12:38.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426618516}) 2024-09-16T03:12:38.984 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.976+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3315371462}) 2024-09-16T03:12:38.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764993498}) 2024-09-16T03:12:38.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=33548022}) 2024-09-16T03:12:38.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764993498}) 2024-09-16T03:12:38.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149043797}) 2024-09-16T03:12:38.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3301121531}) 2024-09-16T03:12:38.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2151729737}) 2024-09-16T03:12:38.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3301121531}) 2024-09-16T03:12:38.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.980+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968610883}) 2024-09-16T03:12:38.988 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.988 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107889991}) 2024-09-16T03:12:38.988 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2013670293}) 2024-09-16T03:12:38.988 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.988 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107889991}) 2024-09-16T03:12:38.988 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470006450}) 2024-09-16T03:12:38.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005660575}) 2024-09-16T03:12:38.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2670354155}) 2024-09-16T03:12:38.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005660575}) 2024-09-16T03:12:38.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.984+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022462915}) 2024-09-16T03:12:38.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541557109}) 2024-09-16T03:12:38.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=426996983}) 2024-09-16T03:12:38.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541557109}) 2024-09-16T03:12:38.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3301226664}) 2024-09-16T03:12:38.993 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.993 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2869626363}) 2024-09-16T03:12:38.993 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1473425671}) 2024-09-16T03:12:38.993 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.993 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2869626363}) 2024-09-16T03:12:38.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2957043984}) 2024-09-16T03:12:38.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851881371}) 2024-09-16T03:12:38.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263554830}) 2024-09-16T03:12:38.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851881371}) 2024-09-16T03:12:38.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.988+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647344122}) 2024-09-16T03:12:38.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019618576}) 2024-09-16T03:12:38.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167431539}) 2024-09-16T03:12:38.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019618576}) 2024-09-16T03:12:38.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2937942918}) 2024-09-16T03:12:38.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021057242}) 2024-09-16T03:12:38.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=445255244}) 2024-09-16T03:12:38.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:38.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021057242}) 2024-09-16T03:12:38.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.992+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=374846334}) 2024-09-16T03:12:39.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529248392}) 2024-09-16T03:12:39.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3177635145}) 2024-09-16T03:12:39.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529248392}) 2024-09-16T03:12:39.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=762369812}) 2024-09-16T03:12:39.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808990648}) 2024-09-16T03:12:39.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4167927839}) 2024-09-16T03:12:39.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808990648}) 2024-09-16T03:12:39.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:38.996+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173048150}) 2024-09-16T03:12:39.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=18,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=5} crc {logm=392612497,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=3028349033,mon_config_key=1833363488,monmap=127340147,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1740207663,osd_snap=2986192999,osdmap=1453783783}) 2024-09-16T03:12:39.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=18,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=7} crc {logm=1241353324,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=3028349033,mon_config_key=486073190,monmap=127340147,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1740207663,osd_snap=1313869725,osdmap=3027684356}) 2024-09-16T03:12:39.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=18,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=5} crc {logm=392612497,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=3028349033,mon_config_key=1833363488,monmap=127340147,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1740207663,osd_snap=2986192999,osdmap=1453783783}) 2024-09-16T03:12:39.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=14,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=18,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=9} crc {logm=2231888828,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=3028349033,mon_config_key=615805402,monmap=3372108098,nvmeofgw=598337686,osd_metadata=773870977,osd_pg_creating=1740207663,osd_snap=660271266,osdmap=3834418977}) 2024-09-16T03:12:39.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2229203159}) 2024-09-16T03:12:39.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=256136950}) 2024-09-16T03:12:39.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2229203159}) 2024-09-16T03:12:39.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.000+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1729101044}) 2024-09-16T03:12:39.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1564982104}) 2024-09-16T03:12:39.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2241064217}) 2024-09-16T03:12:39.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1564982104}) 2024-09-16T03:12:39.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1637867280}) 2024-09-16T03:12:39.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=695877672}) 2024-09-16T03:12:39.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3887450301}) 2024-09-16T03:12:39.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=695877672}) 2024-09-16T03:12:39.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.004+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2621065047}) 2024-09-16T03:12:39.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.008+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.008+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=34141808}) 2024-09-16T03:12:39.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.008+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1480860951}) 2024-09-16T03:12:39.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.008+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.008+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=34141808}) 2024-09-16T03:12:39.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.008+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1258428118}) 2024-09-16T03:12:39.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.028+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.028+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3758049948}) 2024-09-16T03:12:39.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.028+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=99} crc {osdmap=3055210968}) 2024-09-16T03:12:39.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.028+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:12:39.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.028+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3758049948}) 2024-09-16T03:12:39.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:12:39.028+0000 7f260254c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=95} crc {osdmap=1713904565}) 2024-09-16T03:12:39.957 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:12:39.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:12:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient: get_monmap_and_config 2024-09-16T03:12:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient: build_initial_monmap 2024-09-16T03:12:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient: monmap: 2024-09-16T03:12:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:40.093755+0000 2024-09-16T03:12:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:40.093755+0000 2024-09-16T03:12:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:40.094 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 1 Processor -- start 2024-09-16T03:12:40.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 1 -- start start 2024-09-16T03:12:40.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient: init 2024-09-16T03:12:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient(hunting): picked mon.noname-c con 0x7f8b70135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient(hunting): picked mon.noname-a con 0x7f8b7009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient(hunting): picked mon.noname-b con 0x7f8b7013c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b7009f7e0 con 0x7f8b7009fc40 2024-09-16T03:12:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b7009b160 con 0x7f8b70135950 2024-09-16T03:12:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b7009b440 con 0x7f8b7013c010 2024-09-16T03:12:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b751ce640 10 monclient(hunting): authenticate will time out at 1450.743269s 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6e59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8b7009fc40 0x7f8b7013a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57060/0 (socket says 172.21.15.45:57060) 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6e59c640 1 -- 172.21.15.45:0/1079525833 learned_addr learned my addr 172.21.15.45:0/1079525833 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443003252 0 0) 0x7f8b7009f7e0 con 0x7f8b7009fc40 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 10 monclient(hunting): my global_id is 104469 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b58003530 con 0x7f8b7009fc40 2024-09-16T03:12:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464064851 0 0) 0x7f8b7009b440 con 0x7f8b7013c010 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 10 monclient(hunting): my global_id is 124130 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.088+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b7009f7e0 con 0x7f8b7013c010 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2881427517 0 0) 0x7f8b58003530 con 0x7f8b7009fc40 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 >> v1:172.21.15.45:6790/0 conn(0x7f8b7013c010 legacy=0x7f8b7013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 >> v1:172.21.15.46:6789/0 conn(0x7f8b70135950 legacy=0x7f8b70135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: _finish_hunting 0 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 1 monclient: found mon.noname-a 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b70136440 con 0x7f8b7009fc40 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: _finish_auth 0 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: _check_auth_tickets 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8b640016a0 con 0x7f8b7009fc40 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: dump: 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.101 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 5 monclient: authenticate success, global_id 104469 2024-09-16T03:12:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b64001840 con 0x7f8b7009fc40 2024-09-16T03:12:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 1 -- 172.21.15.45:0/1079525833 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8b64001c90 con 0x7f8b7009fc40 2024-09-16T03:12:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6dd9b640 10 monclient: dump: 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 4 set_mon_vals no callback set 2024-09-16T03:12:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: shutdown 2024-09-16T03:12:40.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:40.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- 172.21.15.45:0/1079525833 >> v1:172.21.15.45:6789/0 conn(0x7f8b7009fc40 legacy=0x7f8b7013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- 172.21.15.45:0/1079525833 shutdown_connections 2024-09-16T03:12:40.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- 172.21.15.45:0/1079525833 >> 172.21.15.45:0/1079525833 conn(0x7f8b700a2bd0 msgr2=0x7f8b700a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:40.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- 172.21.15.45:0/1079525833 shutdown_connections 2024-09-16T03:12:40.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- 172.21.15.45:0/1079525833 wait complete. 2024-09-16T03:12:40.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: build_initial_monmap 2024-09-16T03:12:40.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: monmap: 2024-09-16T03:12:40.107 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:40.107 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:40.107 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:40.107 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 librados: starting msgr at 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 librados: starting objecter 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 Processor -- start 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- start start 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 librados: setting wanted keys 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 librados: calling monclient init 2024-09-16T03:12:40.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: init 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient(hunting): picked mon.noname-c con 0x7f8b7009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient(hunting): picked mon.noname-a con 0x7f8b70135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient(hunting): picked mon.noname-b con 0x7f8b7013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b70136fb0 con 0x7f8b70135950 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b70137290 con 0x7f8b7013c010 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b70098a40 con 0x7f8b7009fc40 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6ed9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8b7009fc40 0x7f8b701d0e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40672/0 (socket says 172.21.15.45:40672) 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6ed9d640 1 -- 172.21.15.45:0/2804127062 learned_addr learned my addr 172.21.15.45:0/2804127062 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient(hunting): authenticate will time out at 1450.746436s 2024-09-16T03:12:40.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 client.?.objecter ms_handle_connect 0x7f8b70135950 2024-09-16T03:12:40.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 client.?.objecter ms_handle_connect 0x7f8b7009fc40 2024-09-16T03:12:40.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234262815 0 0) 0x7f8b70098a40 con 0x7f8b7009fc40 2024-09-16T03:12:40.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): my global_id is 124133 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b5c004ec0 con 0x7f8b7009fc40 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2501657646 0 0) 0x7f8b70136fb0 con 0x7f8b70135950 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): my global_id is 104472 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b70098a40 con 0x7f8b70135950 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 client.?.objecter ms_handle_connect 0x7f8b7013c010 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1143011026 0 0) 0x7f8b5c004ec0 con 0x7f8b7009fc40 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b70136fb0 con 0x7f8b7009fc40 2024-09-16T03:12:40.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8b60001a40 con 0x7f8b7009fc40 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): dump: 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2798070749 0 0) 0x7f8b70098a40 con 0x7f8b70135950 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b5c004ec0 con 0x7f8b70135950 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8b64001bd0 con 0x7f8b70135950 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): dump: 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.114 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3493227533 0 0) 0x7f8b70137290 con 0x7f8b7013c010 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient(hunting): my global_id is 22957 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b70098a40 con 0x7f8b7013c010 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2578736187 0 0) 0x7f8b70136fb0 con 0x7f8b7009fc40 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 >> v1:172.21.15.46:6789/0 conn(0x7f8b7013c010 legacy=0x7f8b701d9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 >> v1:172.21.15.45:6789/0 conn(0x7f8b70135950 legacy=0x7f8b701d6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient: _finish_hunting 0 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 monclient: found mon.c 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b70099630 con 0x7f8b7009fc40 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient: _finish_auth 0 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient: _check_auth_tickets 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 5 monclient: authenticate success, global_id 124133 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: _renew_subs 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b701366a0 con 0x7f8b7009fc40 2024-09-16T03:12:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 client.124133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: _renew_subs 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b70097fd0 con 0x7f8b7009fc40 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b751ce640 1 librados: init done 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b60001d40 con 0x7f8b7009fc40 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8b60003a70 con 0x7f8b7009fc40 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b577fe640 10 monclient: dump: 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.092+0000 7f8b6d59a640 4 set_mon_vals no callback set 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b751ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b751ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b751ce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b50005090 con 0x7f8b7009fc40 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f8b60003c10 con 0x7f8b7009fc40 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(343..343 src has 1..343) ==== 5386+0+0 (unknown 2715816078 0 0) 0x7f8b6004b880 con 0x7f8b7009fc40 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b577fe640 10 client.124133.objecter ms_dispatch 0x7f8b70000ba0 osd_map(343..343 src has 1..343) 2024-09-16T03:12:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b577fe640 3 client.124133.objecter handle_osd_map got epochs [343,343] > 0 2024-09-16T03:12:40.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b577fe640 3 client.124133.objecter handle_osd_map decoding full epoch 343 2024-09-16T03:12:40.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b577fe640 20 client.124133.objecter dump_active .. 0 homeless 2024-09-16T03:12:40.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b6e59c640 10 monclient: get_auth_request con 0x7f8b5c03e210 auth_method 0 2024-09-16T03:12:40.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.096+0000 7f8b577fe640 10 client.124133.objecter ms_handle_connect 0x7f8b5c03e210 2024-09-16T03:12:40.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.100+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f8b60016b80 con 0x7f8b7009fc40 2024-09-16T03:12:40.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.100+0000 7f8b577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:40.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.100+0000 7f8b577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:40.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.332+0000 7f8b751ce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:12:40.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.332+0000 7f8b751ce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:40.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.332+0000 7f8b751ce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:40.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.332+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8b50005380 con 0x7f8b7009fc40 2024-09-16T03:12:40.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b577fe640 1 -- 172.21.15.45:0/2804127062 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 3627783513) 0x7f8b60016480 con 0x7f8b7009fc40 2024-09-16T03:12:40.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:40.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b577fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:40.340 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:12:40.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 10 librados: watch_flush enter 2024-09-16T03:12:40.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 10 librados: watch_flush exit 2024-09-16T03:12:40.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 20 client.124133.objecter shutdown clearing up homeless session... 2024-09-16T03:12:40.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 10 client.124133.objecter successfully canceled tick 2024-09-16T03:12:40.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 >> v1:172.21.15.46:6812/149360280 conn(0x7f8b5c03e210 legacy=0x7f8b5c0406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 10 monclient: shutdown 2024-09-16T03:12:40.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 >> v1:172.21.15.45:6790/0 conn(0x7f8b7009fc40 legacy=0x7f8b701d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 shutdown_connections 2024-09-16T03:12:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 >> 172.21.15.45:0/2804127062 conn(0x7f8b700a2bd0 msgr2=0x7f8b700a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 shutdown_connections 2024-09-16T03:12:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 1 -- 172.21.15.45:0/2804127062 wait complete. 2024-09-16T03:12:40.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.336+0000 7f8b751ce640 1 librados: shutdown 2024-09-16T03:12:40.358 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:12:40.358 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:12:40.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:12:40.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:12:40.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient: get_monmap_and_config 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient: build_initial_monmap 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient: monmap: 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:40.485255+0000 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:40.485255+0000 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:40.485 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 1 Processor -- start 2024-09-16T03:12:40.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 1 -- start start 2024-09-16T03:12:40.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient: init 2024-09-16T03:12:40.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:40.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:12:40.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient(hunting): picked mon.noname-a con 0x7f8dc409b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient(hunting): picked mon.noname-c con 0x7f8dc409b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc413b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc409cc20 con 0x7f8dc409b8e0 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc409d820 con 0x7f8dc409b250 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc409b6a0 con 0x7f8dc413b000 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient(hunting): authenticate will time out at 1451.134439s 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc942d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8dc413b000 0x7f8dc413d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38190/0 (socket says 172.21.15.45:38190) 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc942d640 1 -- 172.21.15.45:0/3559681352 learned_addr learned my addr 172.21.15.45:0/3559681352 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2643496136 0 0) 0x7f8dc409b6a0 con 0x7f8dc413b000 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): my global_id is 124136 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dac003530 con 0x7f8dc413b000 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3458741741 0 0) 0x7f8dc409cc20 con 0x7f8dc409b8e0 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): my global_id is 104475 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dc409b6a0 con 0x7f8dc409b8e0 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2128479845 0 0) 0x7f8dc409d820 con 0x7f8dc409b250 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient(hunting): my global_id is 22960 2024-09-16T03:12:40.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dc409cc20 con 0x7f8dc409b250 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3470612378 0 0) 0x7f8dc409b6a0 con 0x7f8dc409b8e0 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 >> v1:172.21.15.45:6790/0 conn(0x7f8dc413b000 legacy=0x7f8dc413d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 >> v1:172.21.15.46:6789/0 conn(0x7f8dc409b250 legacy=0x7f8dc41398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 monclient: found mon.noname-a 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc40a21f0 con 0x7f8dc409b8e0 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: _finish_auth 0 2024-09-16T03:12:40.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:12:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8db4001680 con 0x7f8dc409b8e0 2024-09-16T03:12:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 5 monclient: authenticate success, global_id 104475 2024-09-16T03:12:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: dump: 2024-09-16T03:12:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:40.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8db4001820 con 0x7f8dc409b8e0 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 1 -- 172.21.15.45:0/3559681352 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8db4001c30 con 0x7f8dc409b8e0 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dcae90640 4 set_mon_vals no callback set 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.480+0000 7f8dc37fe640 10 monclient: dump: 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.500 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: shutdown 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- 172.21.15.45:0/3559681352 >> v1:172.21.15.45:6789/0 conn(0x7f8dc409b8e0 legacy=0x7f8dc409bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- 172.21.15.45:0/3559681352 shutdown_connections 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- 172.21.15.45:0/3559681352 >> 172.21.15.45:0/3559681352 conn(0x7f8dc40a2bd0 msgr2=0x7f8dc40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- 172.21.15.45:0/3559681352 shutdown_connections 2024-09-16T03:12:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- 172.21.15.45:0/3559681352 wait complete. 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: build_initial_monmap 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: monmap: 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 librados: starting msgr at 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 librados: starting objecter 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 Processor -- start 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- start start 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 librados: setting wanted keys 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 librados: calling monclient init 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: init 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc409b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient(hunting): picked mon.noname-a con 0x7f8dc409b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:40.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient(hunting): picked mon.noname-c con 0x7f8dc413b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc409a9e0 con 0x7f8dc409b8e0 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc4093fd0 con 0x7f8dc409b250 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc40942b0 con 0x7f8dc413b000 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient(hunting): authenticate will time out at 1451.137765s 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc942d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8dc413b000 0x7f8dc41d4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38192/0 (socket says 172.21.15.45:38192) 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc942d640 1 -- 172.21.15.45:0/4254480683 learned_addr learned my addr 172.21.15.45:0/4254480683 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc3fff640 1 --1- 172.21.15.45:0/4254480683 >> v1:172.21.15.45:6789/0 conn(0x7f8dc409b8e0 0x7f8dc40916d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60544/0 (socket says 172.21.15.45:60544) 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 client.?.objecter ms_handle_connect 0x7f8dc413b000 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 client.?.objecter ms_handle_connect 0x7f8dc409b8e0 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2823833636 0 0) 0x7f8dc40942b0 con 0x7f8dc413b000 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): my global_id is 124139 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8db0004ec0 con 0x7f8dc413b000 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1343901601 0 0) 0x7f8dc409a9e0 con 0x7f8dc409b8e0 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): my global_id is 104478 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dc40942b0 con 0x7f8dc409b8e0 2024-09-16T03:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 client.?.objecter ms_handle_connect 0x7f8dc409b250 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 540466916 0 0) 0x7f8db0004ec0 con 0x7f8dc413b000 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dc409a9e0 con 0x7f8dc413b000 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8dbc0016a0 con 0x7f8dc413b000 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): dump: 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3029504210 0 0) 0x7f8dc40942b0 con 0x7f8dc409b8e0 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8db0004ec0 con 0x7f8dc409b8e0 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8db8001960 con 0x7f8dc409b8e0 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): dump: 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.507 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3004478729 0 0) 0x7f8dc4093fd0 con 0x7f8dc409b250 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): my global_id is 22963 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dc40942b0 con 0x7f8dc409b250 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4244483486 0 0) 0x7f8dc409a9e0 con 0x7f8dc413b000 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 >> v1:172.21.15.46:6789/0 conn(0x7f8dc409b250 legacy=0x7f8dc4090fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 >> v1:172.21.15.45:6789/0 conn(0x7f8dc409b8e0 legacy=0x7f8dc40916d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient: _finish_hunting 0 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 monclient: found mon.c 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc4093ba0 con 0x7f8dc413b000 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient: _finish_auth 0 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:12:40.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 5 monclient: authenticate success, global_id 124139 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: _renew_subs 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc4092590 con 0x7f8dc413b000 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 client.124139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: _renew_subs 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8dc4091fb0 con 0x7f8dc413b000 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 librados: init done 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8dbc004b70 con 0x7f8dc413b000 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8dbc004f00 con 0x7f8dc413b000 2024-09-16T03:12:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc2ffd640 4 set_mon_vals no callback set 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient: dump: 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8da40065d0 con 0x7f8da4002870 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f8dbc010f70 con 0x7f8dc413b000 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 client.124139.objecter ms_handle_connect 0x7f8da4002870 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 client.124139.objecter resend_mon_ops 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(343..343 src has 1..343) ==== 5386+0+0 (unknown 2715816078 0 0) 0x7f8dbc04bb20 con 0x7f8dc413b000 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 client.124139.objecter ms_dispatch 0x7f8dc4000ba0 osd_map(343..343 src has 1..343) 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 3 client.124139.objecter handle_osd_map got epochs [343,343] > 0 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 3 client.124139.objecter handle_osd_map decoding full epoch 343 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 20 client.124139.objecter dump_active .. 0 homeless 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2439097064 0 0) 0x7f8da40065d0 con 0x7f8da4002870 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.484+0000 7f8dc17fa640 10 monclient(hunting): my global_id is 104481 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dc409a9e0 con 0x7f8da4002870 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc3fff640 10 monclient: get_auth_request con 0x7f8db003e210 auth_method 0 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2542632999 0 0) 0x7f8dc409a9e0 con 0x7f8da4002870 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8da40065d0 con 0x7f8da4002870 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8db40015b0 con 0x7f8da4002870 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 10 monclient: dump: 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.513 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3049955433 0 0) 0x7f8da40065d0 con 0x7f8da4002870 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8da400a170 con 0x7f8da4002870 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 10 client.124139.objecter ms_handle_connect 0x7f8db003e210 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8db40018f0 con 0x7f8da4002870 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 >> v1:172.21.15.45:6789/0 conn(0x7f8da4002870 legacy=0x7f8da4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.488+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 >> v1:172.21.15.45:6789/0 conn(0x7f8da4002870 legacy=0x7f8da4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dcae90640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8da4009e30 con 0x7f8da40090f0 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 client.124139.objecter ms_handle_connect 0x7f8da40090f0 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 client.124139.objecter resend_mon_ops 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1978036158 0 0) 0x7f8da4009e30 con 0x7f8da40090f0 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 monclient(hunting): my global_id is 104484 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8da40065d0 con 0x7f8da40090f0 2024-09-16T03:12:40.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1176726383 0 0) 0x7f8da40065d0 con 0x7f8da40090f0 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8da4009e30 con 0x7f8da40090f0 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8db40036c0 con 0x7f8da40090f0 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 monclient: dump: 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.517 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.518 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.518 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1625643242 0 0) 0x7f8da4009e30 con 0x7f8da40090f0 2024-09-16T03:12:40.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8da400c400 con 0x7f8da40090f0 2024-09-16T03:12:40.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4137 (unknown 0 0 403189683) 0x7f8db40036c0 con 0x7f8da40090f0 2024-09-16T03:12:40.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:12:40.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:40.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 >> v1:172.21.15.45:6789/0 conn(0x7f8da40090f0 legacy=0x7f8da4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.492+0000 7f8dc17fa640 1 -- 172.21.15.45:0/4254480683 >> v1:172.21.15.45:6789/0 conn(0x7f8da40090f0 legacy=0x7f8da4009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:40.519 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 32, 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 219593, 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.520 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:40.521 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:40.522 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:12:40.523 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:40.524 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:40.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.526 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:40.527 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:40.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:40.531 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:40.531 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:12:40.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.531 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.531 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:12:40.531 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 10 librados: watch_flush enter 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 10 librados: watch_flush exit 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 20 client.124139.objecter shutdown clearing up homeless session... 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 10 client.124139.objecter successfully canceled tick 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 >> v1:172.21.15.46:6812/149360280 conn(0x7f8db003e210 legacy=0x7f8db00406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 10 monclient: shutdown 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 >> v1:172.21.15.45:6790/0 conn(0x7f8dc413b000 legacy=0x7f8dc41d4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dc942d640 1 -- 172.21.15.45:0/4254480683 reap_dead start 2024-09-16T03:12:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dc942d640 1 -- 172.21.15.45:0/4254480683 reap_dead start 2024-09-16T03:12:40.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 shutdown_connections 2024-09-16T03:12:40.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 >> 172.21.15.45:0/4254480683 conn(0x7f8dc40a2bd0 msgr2=0x7f8dc40a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:40.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 shutdown_connections 2024-09-16T03:12:40.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 1 -- 172.21.15.45:0/4254480683 wait complete. 2024-09-16T03:12:40.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.496+0000 7f8dcae90640 1 librados: shutdown 2024-09-16T03:12:40.534 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:12:40.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:12:40.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:12:40.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.648+0000 7f93c400b640 10 monclient: get_monmap_and_config 2024-09-16T03:12:40.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.648+0000 7f93c400b640 10 monclient: build_initial_monmap 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.648+0000 7f93c400b640 10 monclient: monmap: 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:40.654969+0000 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:40.654969+0000 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:40.655 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.648+0000 7f93c400b640 1 Processor -- start 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.648+0000 7f93c400b640 1 -- start start 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.648+0000 7f93c400b640 10 monclient: init 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): picked mon.noname-b con 0x7f93bc137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): picked mon.noname-a con 0x7f93bc138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): picked mon.noname-c con 0x7f93bc13e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93bc0a4f90 con 0x7f93bc138100 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93bc0a4cb0 con 0x7f93bc13e2a0 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93bc09dc70 con 0x7f93bc137d20 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:40.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): authenticate will time out at 1451.304517s 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c1da7640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f93bc137d20 0x7f93bc0a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38194/0 (socket says 172.21.15.45:38194) 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c1da7640 1 -- 172.21.15.45:0/477709323 learned_addr learned my addr 172.21.15.45:0/477709323 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2152225409 0 0) 0x7f93bc0a4f90 con 0x7f93bc138100 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): my global_id is 104487 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a8003530 con 0x7f93bc138100 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492522123 0 0) 0x7f93bc09dc70 con 0x7f93bc137d20 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): my global_id is 124142 2024-09-16T03:12:40.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93bc0a4f90 con 0x7f93bc137d20 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975407653 0 0) 0x7f93bc0a4cb0 con 0x7f93bc13e2a0 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient(hunting): my global_id is 22966 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93bc09dc70 con 0x7f93bc13e2a0 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 419831515 0 0) 0x7f93a8003530 con 0x7f93bc138100 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 >> v1:172.21.15.45:6790/0 conn(0x7f93bc137d20 legacy=0x7f93bc0a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 >> v1:172.21.15.46:6789/0 conn(0x7f93bc13e2a0 legacy=0x7f93bc140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: _finish_hunting 0 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 monclient: found mon.noname-a 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:40.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93bc09cc50 con 0x7f93bc138100 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: _finish_auth 0 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: _check_auth_tickets 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f93ac0016a0 con 0x7f93bc138100 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: dump: 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 5 monclient: authenticate success, global_id 104487 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93ac001840 con 0x7f93bc138100 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 1 -- 172.21.15.45:0/477709323 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f93ac001cd0 con 0x7f93bc138100 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c0da5640 10 monclient: dump: 2024-09-16T03:12:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 4 set_mon_vals no callback set 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient: shutdown 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- 172.21.15.45:0/477709323 >> v1:172.21.15.45:6789/0 conn(0x7f93bc138100 legacy=0x7f93bc13cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- 172.21.15.45:0/477709323 shutdown_connections 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- 172.21.15.45:0/477709323 >> 172.21.15.45:0/477709323 conn(0x7f93bc0aa110 msgr2=0x7f93bc0aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- 172.21.15.45:0/477709323 shutdown_connections 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- 172.21.15.45:0/477709323 wait complete. 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient: build_initial_monmap 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient: monmap: 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:40.670 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 librados: starting msgr at 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 librados: starting objecter 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 Processor -- start 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- start start 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 librados: setting wanted keys 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 librados: calling monclient init 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient: init 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:12:40.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): picked mon.noname-c con 0x7f93bc137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): picked mon.noname-b con 0x7f93bc138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): picked mon.noname-a con 0x7f93bc13e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93bc13c0a0 con 0x7f93bc13e2a0 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93bc139770 con 0x7f93bc138100 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93bc0a1300 con 0x7f93bc137d20 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c400b640 10 monclient(hunting): authenticate will time out at 1451.307801s 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c1da7640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f93bc137d20 0x7f93bc138950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38210/0 (socket says 172.21.15.45:38210) 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c25a8640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f93bc13e2a0 0x7f93bc1d4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60580/0 (socket says 172.21.15.45:60580) 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.652+0000 7f93c1da7640 1 -- 172.21.15.45:0/258854027 learned_addr learned my addr 172.21.15.45:0/258854027 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.?.objecter ms_handle_connect 0x7f93bc137d20 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.?.objecter ms_handle_connect 0x7f93bc13e2a0 2024-09-16T03:12:40.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471941029 0 0) 0x7f93bc0a1300 con 0x7f93bc137d20 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): my global_id is 124145 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a0004ec0 con 0x7f93bc137d20 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1266097454 0 0) 0x7f93bc13c0a0 con 0x7f93bc13e2a0 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): my global_id is 104490 2024-09-16T03:12:40.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93bc0a1300 con 0x7f93bc13e2a0 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.?.objecter ms_handle_connect 0x7f93bc138100 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1407380205 0 0) 0x7f93bc0a1300 con 0x7f93bc13e2a0 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93bc13c0a0 con 0x7f93bc13e2a0 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1784289456 0 0) 0x7f93a0004ec0 con 0x7f93bc137d20 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93bc0a1300 con 0x7f93bc137d20 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f93b4001c40 con 0x7f93bc13e2a0 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): dump: 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f93b8001900 con 0x7f93bc137d20 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): dump: 2024-09-16T03:12:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836946416 0 0) 0x7f93bc139770 con 0x7f93bc138100 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): my global_id is 22969 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a0004ec0 con 0x7f93bc138100 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1925234103 0 0) 0x7f93bc13c0a0 con 0x7f93bc13e2a0 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 >> v1:172.21.15.45:6790/0 conn(0x7f93bc137d20 legacy=0x7f93bc138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 >> v1:172.21.15.46:6789/0 conn(0x7f93bc138100 legacy=0x7f93bc139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: _finish_hunting 0 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 monclient: found mon.a 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93bc13b540 con 0x7f93bc13e2a0 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: _finish_auth 0 2024-09-16T03:12:40.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 5 monclient: authenticate success, global_id 104490 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 10 monclient: _renew_subs 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93bc13bdc0 con 0x7f93bc13e2a0 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 10 client.104490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 10 monclient: _renew_subs 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f93bc0a4210 con 0x7f93bc13e2a0 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 1 librados: init done 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93b40039a0 con 0x7f93bc13e2a0 2024-09-16T03:12:40.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f93b4003d70 con 0x7f93bc13e2a0 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b37fe640 4 set_mon_vals no callback set 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: dump: 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.679 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f93b4003f90 con 0x7f93bc13e2a0 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f93a40065d0 con 0x7f93a4002870 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(343..343 src has 1..343) ==== 5386+0+0 (unknown 2715816078 0 0) 0x7f93b404ba20 con 0x7f93bc13e2a0 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.104490.objecter ms_dispatch 0x7f93bc000ba0 osd_map(343..343 src has 1..343) 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 3 client.104490.objecter handle_osd_map got epochs [343,343] > 0 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 3 client.104490.objecter handle_osd_map decoding full epoch 343 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 20 client.104490.objecter dump_active .. 0 homeless 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.104490.objecter ms_handle_connect 0x7f93a4002870 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.104490.objecter resend_mon_ops 2024-09-16T03:12:40.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948667750 0 0) 0x7f93a40065d0 con 0x7f93a4002870 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient(hunting): my global_id is 124148 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93bc13c0a0 con 0x7f93a4002870 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93c15a6640 10 monclient: get_auth_request con 0x7f93a003e2e0 auth_method 0 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 254749311 0 0) 0x7f93bc13c0a0 con 0x7f93a4002870 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93a40065d0 con 0x7f93a4002870 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f93b8004990 con 0x7f93a4002870 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: dump: 2024-09-16T03:12:40.681 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2310148790 0 0) 0x7f93a40065d0 con 0x7f93a4002870 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93a400a170 con 0x7f93a4002870 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 client.104490.objecter ms_handle_connect 0x7f93a003e2e0 2024-09-16T03:12:40.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f93b8006bd0 con 0x7f93a4002870 2024-09-16T03:12:40.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:40.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:40.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 >> v1:172.21.15.45:6790/0 conn(0x7f93a4002870 legacy=0x7f93a4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.656+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 >> v1:172.21.15.45:6790/0 conn(0x7f93a4002870 legacy=0x7f93a4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:40.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:12:40.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f93a4009e10 con 0x7f93a40090f0 2024-09-16T03:12:40.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 client.104490.objecter ms_handle_connect 0x7f93a40090f0 2024-09-16T03:12:40.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 client.104490.objecter resend_mon_ops 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1324075652 0 0) 0x7f93a4009e10 con 0x7f93a40090f0 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 monclient(hunting): my global_id is 124151 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a40065d0 con 0x7f93a40090f0 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1427060207 0 0) 0x7f93a40065d0 con 0x7f93a40090f0 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93a4009e10 con 0x7f93a40090f0 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f93b8006360 con 0x7f93a40090f0 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 monclient: dump: 2024-09-16T03:12:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 407100780 0 0) 0x7f93a4009e10 con 0x7f93a40090f0 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f93a400c3b0 con 0x7f93a40090f0 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3966 (unknown 0 0 934741232) 0x7f93b8006180 con 0x7f93a40090f0 2024-09-16T03:12:40.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 >> v1:172.21.15.45:6790/0 conn(0x7f93a40090f0 legacy=0x7f93a4009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93b27fc640 1 -- 172.21.15.45:0/258854027 >> v1:172.21.15.45:6790/0 conn(0x7f93a40090f0 legacy=0x7f93a4009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 32, 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 2895, 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:12:40.687 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:40.688 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:40.689 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:12:40.690 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:40.691 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:40.692 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:40.692 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:40.692 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:40.692 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:40.692 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:40.692 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:40.692 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:40.692 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:40.692 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:12:40.693 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:40.694 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:12:40.696 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.697 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:12:40.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.698 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:40.698 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:40.698 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:40.698 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.698 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.698 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:12:40.698 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:12:40.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 10 librados: watch_flush enter 2024-09-16T03:12:40.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 10 librados: watch_flush exit 2024-09-16T03:12:40.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 20 client.104490.objecter shutdown clearing up homeless session... 2024-09-16T03:12:40.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 10 client.104490.objecter successfully canceled tick 2024-09-16T03:12:40.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 >> v1:172.21.15.46:6812/149360280 conn(0x7f93a003e2e0 legacy=0x7f93a00407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 10 monclient: shutdown 2024-09-16T03:12:40.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:40.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 >> v1:172.21.15.45:6789/0 conn(0x7f93bc13e2a0 legacy=0x7f93bc1d4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c25a8640 1 -- 172.21.15.45:0/258854027 reap_dead start 2024-09-16T03:12:40.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c25a8640 1 -- 172.21.15.45:0/258854027 reap_dead start 2024-09-16T03:12:40.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 shutdown_connections 2024-09-16T03:12:40.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 >> 172.21.15.45:0/258854027 conn(0x7f93bc0aa110 msgr2=0x7f93bc0742d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:40.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 shutdown_connections 2024-09-16T03:12:40.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 1 -- 172.21.15.45:0/258854027 wait complete. 2024-09-16T03:12:40.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.664+0000 7f93c400b640 1 librados: shutdown 2024-09-16T03:12:40.701 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:12:40.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:12:40.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:12:40.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient: get_monmap_and_config 2024-09-16T03:12:40.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient: build_initial_monmap 2024-09-16T03:12:40.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient: monmap: 2024-09-16T03:12:40.820 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:40.820714+0000 2024-09-16T03:12:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:40.820714+0000 2024-09-16T03:12:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:40.821 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:40.821 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 Processor -- start 2024-09-16T03:12:40.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 -- start start 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient: init 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient(hunting): picked mon.noname-c con 0x7f3024135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient(hunting): picked mon.noname-a con 0x7f302409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient(hunting): picked mon.noname-b con 0x7f302413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f302409f7e0 con 0x7f302409fc40 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f302409b160 con 0x7f3024135950 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f302409b440 con 0x7f302413c010 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient(hunting): authenticate will time out at 1451.470069s 2024-09-16T03:12:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f3029721640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f302413c010 0x7f302413e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38238/0 (socket says 172.21.15.45:38238) 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f3029721640 1 -- 172.21.15.45:0/621446921 learned_addr learned my addr 172.21.15.45:0/621446921 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2181456195 0 0) 0x7f302409b440 con 0x7f302413c010 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient(hunting): my global_id is 124154 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3004003530 con 0x7f302413c010 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012738066 0 0) 0x7f302409f7e0 con 0x7f302409fc40 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient(hunting): my global_id is 104493 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f302409b440 con 0x7f302409fc40 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2075740245 0 0) 0x7f302409b440 con 0x7f302409fc40 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 >> v1:172.21.15.45:6790/0 conn(0x7f302413c010 legacy=0x7f302413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 >> v1:172.21.15.46:6789/0 conn(0x7f3024135950 legacy=0x7f3024135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: _finish_hunting 0 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 monclient: found mon.noname-a 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3024136440 con 0x7f302409fc40 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: _finish_auth 0 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: _check_auth_tickets 2024-09-16T03:12:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f300c001a00 con 0x7f302409fc40 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: dump: 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 5 monclient: authenticate success, global_id 104493 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f300c001ba0 con 0x7f302409fc40 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 1 -- 172.21.15.45:0/621446921 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f300c002030 con 0x7f302409fc40 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 4 set_mon_vals no callback set 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f30237fe640 10 monclient: dump: 2024-09-16T03:12:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 10 monclient: shutdown 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 -- 172.21.15.45:0/621446921 >> v1:172.21.15.45:6789/0 conn(0x7f302409fc40 legacy=0x7f302413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 -- 172.21.15.45:0/621446921 shutdown_connections 2024-09-16T03:12:40.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 -- 172.21.15.45:0/621446921 >> 172.21.15.45:0/621446921 conn(0x7f30240a2bd0 msgr2=0x7f30240a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 -- 172.21.15.45:0/621446921 shutdown_connections 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.816+0000 7f302b184640 1 -- 172.21.15.45:0/621446921 wait complete. 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient: build_initial_monmap 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient: monmap: 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 librados: starting msgr at 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 librados: starting objecter 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 Processor -- start 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 -- start start 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 librados: setting wanted keys 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 librados: calling monclient init 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient: init 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient(hunting): picked mon.noname-c con 0x7f302409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient(hunting): picked mon.noname-b con 0x7f3024135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:40.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient(hunting): picked mon.noname-a con 0x7f302413c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f302409f5d0 con 0x7f302413c010 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30240742d0 con 0x7f3024135950 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f302409e5d0 con 0x7f302409fc40 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient(hunting): authenticate will time out at 1451.473239s 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f3028f20640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f302409fc40 0x7f30240727c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38246/0 (socket says 172.21.15.45:38246) 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f3028f20640 1 -- 172.21.15.45:0/3371759592 learned_addr learned my addr 172.21.15.45:0/3371759592 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 client.?.objecter ms_handle_connect 0x7f302413c010 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 client.?.objecter ms_handle_connect 0x7f302409fc40 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3180004360 0 0) 0x7f301c001920 con 0x7f302413c010 2024-09-16T03:12:40.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): my global_id is 104496 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3008004ef0 con 0x7f302413c010 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682345333 0 0) 0x7f302409e5d0 con 0x7f302409fc40 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): my global_id is 124157 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f301c001920 con 0x7f302409fc40 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 client.?.objecter ms_handle_connect 0x7f3024135950 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1205459325 0 0) 0x7f3008004ef0 con 0x7f302413c010 2024-09-16T03:12:40.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f302409e5d0 con 0x7f302413c010 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3418160444 0 0) 0x7f301c001920 con 0x7f302409fc40 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3008004ef0 con 0x7f302409fc40 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995557424 0 0) 0x7f30240742d0 con 0x7f3024135950 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): my global_id is 22975 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f301c001920 con 0x7f3024135950 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f301c001bb0 con 0x7f302413c010 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): dump: 2024-09-16T03:12:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3014001980 con 0x7f302409fc40 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient(hunting): dump: 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1744001143 0 0) 0x7f302409e5d0 con 0x7f302413c010 2024-09-16T03:12:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 >> v1:172.21.15.45:6790/0 conn(0x7f302409fc40 legacy=0x7f30240727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 >> v1:172.21.15.46:6789/0 conn(0x7f3024135950 legacy=0x7f3024072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient: _finish_hunting 0 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 monclient: found mon.a 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30240735e0 con 0x7f302413c010 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient: _finish_auth 0 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient: _check_auth_tickets 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 5 monclient: authenticate success, global_id 104496 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient: _renew_subs 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3024073c10 con 0x7f302413c010 2024-09-16T03:12:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 client.104496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient: _renew_subs 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3024074050 con 0x7f302413c010 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 librados: init done 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f301c004aa0 con 0x7f302413c010 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f301c004df0 con 0x7f302413c010 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 monclient: dump: 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.843 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f3022ffd640 4 set_mon_vals no callback set 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f301c010e60 con 0x7f302413c010 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ffc0065d0 con 0x7f2ffc002870 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(343..343 src has 1..343) ==== 5386+0+0 (unknown 2715816078 0 0) 0x7f301c04b760 con 0x7f302413c010 2024-09-16T03:12:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 client.104496.objecter ms_dispatch 0x7f3024000ba0 osd_map(343..343 src has 1..343) 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 3 client.104496.objecter handle_osd_map got epochs [343,343] > 0 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 3 client.104496.objecter handle_osd_map decoding full epoch 343 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 20 client.104496.objecter dump_active .. 0 homeless 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 client.104496.objecter ms_handle_connect 0x7f2ffc002870 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f30217fa640 10 client.104496.objecter resend_mon_ops 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.820+0000 7f3023fff640 10 monclient: get_auth_request con 0x7f300803e310 auth_method 0 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3571665777 0 0) 0x7f2ffc0065d0 con 0x7f2ffc002870 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 monclient(hunting): my global_id is 22978 2024-09-16T03:12:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f302409e5d0 con 0x7f2ffc002870 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 client.104496.objecter ms_handle_connect 0x7f300803e310 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 880769386 0 0) 0x7f302409e5d0 con 0x7f2ffc002870 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ffc0065d0 con 0x7f2ffc002870 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f30140037a0 con 0x7f2ffc002870 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 monclient: dump: 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.847 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.847 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.847 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2023633923 0 0) 0x7f2ffc0065d0 con 0x7f2ffc002870 2024-09-16T03:12:40.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ffc00a170 con 0x7f2ffc002870 2024-09-16T03:12:40.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f30140057b0 con 0x7f2ffc002870 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 >> v1:172.21.15.46:6789/0 conn(0x7f2ffc002870 legacy=0x7f2ffc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.824+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 >> v1:172.21.15.46:6789/0 conn(0x7f2ffc002870 legacy=0x7f2ffc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.828+0000 7f302b184640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.828+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ffc009e40 con 0x7f2ffc009120 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 client.104496.objecter ms_handle_connect 0x7f2ffc009120 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 client.104496.objecter resend_mon_ops 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 611569778 0 0) 0x7f2ffc009e40 con 0x7f2ffc009120 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 monclient(hunting): my global_id is 22981 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ffc0065d0 con 0x7f2ffc009120 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2403954048 0 0) 0x7f2ffc0065d0 con 0x7f2ffc009120 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ffc009e40 con 0x7f2ffc009120 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3014003e90 con 0x7f2ffc009120 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 monclient: dump: 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 276022514 0 0) 0x7f2ffc009e40 con 0x7f2ffc009120 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ffc00c400 con 0x7f2ffc009120 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 3825426505) 0x7f3014005960 con 0x7f2ffc009120 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 >> v1:172.21.15.46:6789/0 conn(0x7f2ffc009120 legacy=0x7f2ffc009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f30217fa640 1 -- 172.21.15.45:0/3371759592 >> v1:172.21.15.46:6789/0 conn(0x7f2ffc009120 legacy=0x7f2ffc009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 32, 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 482642, 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:12:40.852 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:40.853 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:40.854 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:12:40.855 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:40.856 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:40.856 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:12:40.856 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:12:40.856 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:40.857 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:12:40.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.859 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:40.859 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:40.859 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:40.859 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:40.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.859 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:40.859 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:12:40.859 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:40.859 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:40.860 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:40.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.862 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:40.862 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:40.862 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:40.862 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:40.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.862 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:40.862 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.864 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:40.865 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:40.866 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:40.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:40.866 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:12:40.866 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:12:40.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 10 librados: watch_flush enter 2024-09-16T03:12:40.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 10 librados: watch_flush exit 2024-09-16T03:12:40.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 20 client.104496.objecter shutdown clearing up homeless session... 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 10 client.104496.objecter successfully canceled tick 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 >> v1:172.21.15.46:6812/149360280 conn(0x7f300803e310 legacy=0x7f30080407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 10 monclient: shutdown 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 >> v1:172.21.15.45:6789/0 conn(0x7f302413c010 legacy=0x7f30241d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f3029721640 1 -- 172.21.15.45:0/3371759592 reap_dead start 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f3029721640 1 -- 172.21.15.45:0/3371759592 reap_dead start 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 shutdown_connections 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 >> 172.21.15.45:0/3371759592 conn(0x7f30240a2bd0 msgr2=0x7f30240a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 shutdown_connections 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 1 -- 172.21.15.45:0/3371759592 wait complete. 2024-09-16T03:12:40.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.832+0000 7f302b184640 1 librados: shutdown 2024-09-16T03:12:40.869 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:12:40.869 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T03:12:40.869 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:12:40.869 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T03:12:40.869 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-16T03:12:40.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-16T03:12:40.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.984+0000 7fbb41cea640 10 monclient: get_monmap_and_config 2024-09-16T03:12:40.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.984+0000 7fbb41cea640 10 monclient: build_initial_monmap 2024-09-16T03:12:40.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.984+0000 7fbb41cea640 10 monclient: monmap: 2024-09-16T03:12:40.991 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:40.991 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:40.992 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:40.991819+0000 2024-09-16T03:12:40.992 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:40.991819+0000 2024-09-16T03:12:40.992 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:40.992 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:40.992 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:40.992 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:40.992 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:40.992 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:40.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 1 Processor -- start 2024-09-16T03:12:40.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 1 -- start start 2024-09-16T03:12:40.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 10 monclient: init 2024-09-16T03:12:40.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:40.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:12:40.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 10 monclient(hunting): picked mon.noname-b con 0x7fbb3c137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:40.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 10 monclient(hunting): picked mon.noname-c con 0x7fbb3c138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:40.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 10 monclient(hunting): picked mon.noname-a con 0x7fbb3c13e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:40.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c0a4f90 con 0x7fbb3c13e2a0 2024-09-16T03:12:40.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c0a4cb0 con 0x7fbb3c138100 2024-09-16T03:12:40.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c09dc70 con 0x7fbb3c137d20 2024-09-16T03:12:40.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:40.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 10 monclient(hunting): authenticate will time out at 1451.641263s 2024-09-16T03:12:40.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3b7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fbb3c137d20 0x7fbb3c0a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38266/0 (socket says 172.21.15.45:38266) 2024-09-16T03:12:40.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3b7fe640 1 -- 172.21.15.45:0/537545416 learned_addr learned my addr 172.21.15.45:0/537545416 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:40.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1386195955 0 0) 0x7fbb3c0a4f90 con 0x7fbb3c13e2a0 2024-09-16T03:12:40.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:40.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:40.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:40.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient(hunting): my global_id is 104499 2024-09-16T03:12:40.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb28003530 con 0x7fbb3c13e2a0 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2975563031 0 0) 0x7fbb3c09dc70 con 0x7fbb3c137d20 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient(hunting): my global_id is 124160 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb3c0a4f90 con 0x7fbb3c137d20 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3286540202 0 0) 0x7fbb28003530 con 0x7fbb3c13e2a0 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 >> v1:172.21.15.45:6790/0 conn(0x7fbb3c137d20 legacy=0x7fbb3c0a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 >> v1:172.21.15.46:6789/0 conn(0x7fbb3c138100 legacy=0x7fbb3c13cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: _finish_hunting 0 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 monclient: found mon.noname-a 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c09cc50 con 0x7fbb3c13e2a0 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: _finish_auth 0 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: _check_auth_tickets 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 5 monclient: authenticate success, global_id 104499 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb34001950 con 0x7fbb3c13e2a0 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:12:41.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: dump: 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:41.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb34001af0 con 0x7fbb3c13e2a0 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 1 -- 172.21.15.45:0/537545416 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb34001f40 con 0x7fbb3c13e2a0 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb41cea640 4 set_mon_vals no callback set 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.988+0000 7fbb3affd640 10 monclient: dump: 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.009 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: shutdown 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- 172.21.15.45:0/537545416 >> v1:172.21.15.45:6789/0 conn(0x7fbb3c13e2a0 legacy=0x7fbb3c140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- 172.21.15.45:0/537545416 shutdown_connections 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- 172.21.15.45:0/537545416 >> 172.21.15.45:0/537545416 conn(0x7fbb3c0aa110 msgr2=0x7fbb3c0aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- 172.21.15.45:0/537545416 shutdown_connections 2024-09-16T03:12:41.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- 172.21.15.45:0/537545416 wait complete. 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: build_initial_monmap 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: monmap: 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 librados: starting msgr at 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 librados: starting objecter 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 Processor -- start 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- start start 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 librados: setting wanted keys 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 librados: calling monclient init 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: init 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient(hunting): picked mon.noname-a con 0x7fbb3c137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient(hunting): picked mon.noname-c con 0x7fbb3c138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient(hunting): picked mon.noname-b con 0x7fbb3c13e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c13ba80 con 0x7fbb3c137d20 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c13bd60 con 0x7fbb3c13e2a0 2024-09-16T03:12:41.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c139770 con 0x7fbb3c138100 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient(hunting): authenticate will time out at 1451.645943s 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb32ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fbb3c138100 0x7fbb3c139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38272/0 (socket says 172.21.15.45:38272) 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb32ffd640 1 -- 172.21.15.45:0/3163493735 learned_addr learned my addr 172.21.15.45:0/3163493735 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 client.?.objecter ms_handle_connect 0x7fbb3c138100 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 client.?.objecter ms_handle_connect 0x7fbb3c137d20 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950577669 0 0) 0x7fbb3c139770 con 0x7fbb3c138100 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): my global_id is 124163 2024-09-16T03:12:41.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb1c004ec0 con 0x7fbb3c138100 2024-09-16T03:12:41.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531021511 0 0) 0x7fbb3c13ba80 con 0x7fbb3c137d20 2024-09-16T03:12:41.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:12:41.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:41.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:41.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): my global_id is 104502 2024-09-16T03:12:41.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb3c139770 con 0x7fbb3c137d20 2024-09-16T03:12:41.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 client.?.objecter ms_handle_connect 0x7fbb3c13e2a0 2024-09-16T03:12:41.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3589588873 0 0) 0x7fbb1c004ec0 con 0x7fbb3c138100 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb3c13ba80 con 0x7fbb3c138100 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb24001a40 con 0x7fbb3c138100 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): dump: 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1730874350 0 0) 0x7fbb3c139770 con 0x7fbb3c137d20 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb1c004ec0 con 0x7fbb3c137d20 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb2c001820 con 0x7fbb3c137d20 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient(hunting): dump: 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.016 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2674382356 0 0) 0x7fbb3c13ba80 con 0x7fbb3c138100 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 >> v1:172.21.15.46:6789/0 conn(0x7fbb3c13e2a0 legacy=0x7fbb3c1d4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 >> v1:172.21.15.45:6789/0 conn(0x7fbb3c137d20 legacy=0x7fbb3c138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 monclient: found mon.c 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c139f40 con 0x7fbb3c138100 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient: _finish_auth 0 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 5 monclient: authenticate success, global_id 124163 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: _renew_subs 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3c13c040 con 0x7fbb3c138100 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 client.124163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: _renew_subs 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb3c13c7a0 con 0x7fbb3c138100 2024-09-16T03:12:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 librados: init done 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb24001cc0 con 0x7fbb3c138100 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb24003a70 con 0x7fbb3c138100 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb3a7fc640 4 set_mon_vals no callback set 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 10 monclient: dump: 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.019 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbb200065d0 con 0x7fbb20002870 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.992+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fbb24003c10 con 0x7fbb3c138100 2024-09-16T03:12:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 client.124163.objecter ms_handle_connect 0x7fbb20002870 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 client.124163.objecter resend_mon_ops 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(344..344 src has 1..344) ==== 5386+0+0 (unknown 2673046086 0 0) 0x7fbb2404b800 con 0x7fbb3c138100 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 client.124163.objecter ms_dispatch 0x7fbb3c000ba0 osd_map(344..344 src has 1..344) 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 3 client.124163.objecter handle_osd_map got epochs [344,344] > 0 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 3 client.124163.objecter handle_osd_map decoding full epoch 344 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 20 client.124163.objecter dump_active .. 0 homeless 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 872361132 0 0) 0x7fbb200065d0 con 0x7fbb20002870 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 monclient(hunting): my global_id is 104505 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb3c13ba80 con 0x7fbb20002870 2024-09-16T03:12:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb3bfff640 10 monclient: get_auth_request con 0x7fbb1c03e120 auth_method 0 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3208284850 0 0) 0x7fbb3c13ba80 con 0x7fbb20002870 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb200065d0 con 0x7fbb20002870 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb2c0047f0 con 0x7fbb20002870 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 monclient: dump: 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1761832116 0 0) 0x7fbb200065d0 con 0x7fbb20002870 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb2000a170 con 0x7fbb20002870 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 client.124163.objecter ms_handle_connect 0x7fbb1c03e120 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbb2c006a30 con 0x7fbb20002870 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 >> v1:172.21.15.45:6789/0 conn(0x7fbb20002870 legacy=0x7fbb20004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:40.996+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 >> v1:172.21.15.45:6789/0 conn(0x7fbb20002870 legacy=0x7fbb20004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:41.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.000+0000 7fbb41cea640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:12:41.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.000+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbb20009e50 con 0x7fbb20009130 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 client.124163.objecter ms_handle_connect 0x7fbb20009130 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 client.124163.objecter resend_mon_ops 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170010225 0 0) 0x7fbb20009e50 con 0x7fbb20009130 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 monclient(hunting): my global_id is 104508 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb200065d0 con 0x7fbb20009130 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 29285648 0 0) 0x7fbb200065d0 con 0x7fbb20009130 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb20009e50 con 0x7fbb20009130 2024-09-16T03:12:41.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbb2c0060d0 con 0x7fbb20009130 2024-09-16T03:12:41.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:41.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:12:41.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 monclient: dump: 2024-09-16T03:12:41.026 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.026 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.026 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.026 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3380112565 0 0) 0x7fbb20009e50 con 0x7fbb20009130 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fbb2000c400 con 0x7fbb20009130 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (unknown 0 0 2946029456) 0x7fbb2c005fa0 con 0x7fbb20009130 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 >> v1:172.21.15.45:6789/0 conn(0x7fbb20009130 legacy=0x7fbb200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb38ff9640 1 -- 172.21.15.45:0/3163493735 >> v1:172.21.15.45:6789/0 conn(0x7fbb20009130 legacy=0x7fbb200095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:41.028 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:12:41.028 INFO:teuthology.orchestra.run.smithi045.stdout: "ret": 0, 2024-09-16T03:12:41.028 INFO:teuthology.orchestra.run.smithi045.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T03:12:41.028 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:12:41.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 10 librados: watch_flush enter 2024-09-16T03:12:41.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 10 librados: watch_flush exit 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 20 client.124163.objecter shutdown clearing up homeless session... 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 10 client.124163.objecter successfully canceled tick 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 >> v1:172.21.15.46:6812/149360280 conn(0x7fbb1c03e120 legacy=0x7fbb1c0405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 10 monclient: shutdown 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 >> v1:172.21.15.45:6790/0 conn(0x7fbb3c138100 legacy=0x7fbb3c139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb3bfff640 1 -- 172.21.15.45:0/3163493735 reap_dead start 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb3bfff640 1 -- 172.21.15.45:0/3163493735 reap_dead start 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 shutdown_connections 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 >> 172.21.15.45:0/3163493735 conn(0x7fbb3c0aa110 msgr2=0x7fbb3c0a3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 shutdown_connections 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 1 -- 172.21.15.45:0/3163493735 wait complete. 2024-09-16T03:12:41.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.004+0000 7fbb41cea640 1 librados: shutdown 2024-09-16T03:12:41.031 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:12:41.031 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:12:41.031 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:12:41.044 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:12:41.044 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:12:41.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient: get_monmap_and_config 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient: build_initial_monmap 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient: monmap: 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:41.198311+0000 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:41.198311+0000 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.198 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:41.199 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:41.199 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:41.199 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 1 Processor -- start 2024-09-16T03:12:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 1 -- start start 2024-09-16T03:12:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient: init 2024-09-16T03:12:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:41.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient(hunting): picked mon.noname-c con 0x7f41ec135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient(hunting): picked mon.noname-a con 0x7f41ec09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient(hunting): picked mon.noname-b con 0x7f41ec13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec09f7e0 con 0x7f41ec09fc40 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec09b160 con 0x7f41ec135950 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec09b440 con 0x7f41ec13c010 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f41a8640 10 monclient(hunting): authenticate will time out at 1451.847656s 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f1743640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f41ec09fc40 legacy=0x7f41ec13a8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f2745640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f41ec13c010 0x7f41ec13e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38280/0 (socket says 172.21.15.45:38280) 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.192+0000 7f41f2745640 1 -- 172.21.15.45:0/4193774346 learned_addr learned my addr 172.21.15.45:0/4193774346 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 -- 172.21.15.45:0/4193774346 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2872986443 0 0) 0x7f41ec09b440 con 0x7f41ec13c010 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient(hunting): _negotiate 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient(hunting): my global_id is 124166 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 -- 172.21.15.45:0/4193774346 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41d8003530 con 0x7f41ec13c010 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 -- 172.21.15.45:0/4193774346 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1039999907 0 0) 0x7f41d8003530 con 0x7f41ec13c010 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 -- 172.21.15.45:0/4193774346 >> v1:172.21.15.46:6789/0 conn(0x7f41ec135950 legacy=0x7f41ec135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 -- 172.21.15.45:0/4193774346 >> v1:172.21.15.45:6789/0 conn(0x7f41ec09fc40 legacy=0x7f41ec13a8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: _finish_hunting 0 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 monclient: found mon.noname-b 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 -- 172.21.15.45:0/4193774346 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec136440 con 0x7f41ec13c010 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: _finish_auth 0 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: _check_auth_tickets 2024-09-16T03:12:41.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 -- 172.21.15.45:0/4193774346 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f41e4001a20 con 0x7f41ec13c010 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: dump: 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 5 monclient: authenticate success, global_id 124166 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 -- 172.21.15.45:0/4193774346 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41e4001c40 con 0x7f41ec13c010 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 1 -- 172.21.15.45:0/4193774346 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f41e40020d0 con 0x7f41ec13c010 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f0f42640 10 monclient: dump: 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.209 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 4 set_mon_vals no callback set 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: shutdown 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- 172.21.15.45:0/4193774346 >> v1:172.21.15.45:6790/0 conn(0x7f41ec13c010 legacy=0x7f41ec13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- 172.21.15.45:0/4193774346 shutdown_connections 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- 172.21.15.45:0/4193774346 >> 172.21.15.45:0/4193774346 conn(0x7f41ec0a2bd0 msgr2=0x7f41ec0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- 172.21.15.45:0/4193774346 shutdown_connections 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- 172.21.15.45:0/4193774346 wait complete. 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: build_initial_monmap 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: monmap: 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:41.211 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 librados: starting msgr at 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 librados: starting objecter 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 Processor -- start 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- start start 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 librados: setting wanted keys 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 librados: calling monclient init 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: init 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient(hunting): picked mon.noname-b con 0x7f41ec09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient(hunting): picked mon.noname-a con 0x7f41ec135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient(hunting): picked mon.noname-c con 0x7f41ec13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec13bb40 con 0x7f41ec135950 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec071b80 con 0x7f41ec09fc40 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f1743640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f41ec135950 legacy=0x7f41ec06ed20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec0712e0 con 0x7f41ec13c010 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient(hunting): authenticate will time out at 1451.850721s 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f2745640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f41ec13c010 0x7f41ec1d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38284/0 (socket says 172.21.15.45:38284) 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f2745640 1 -- 172.21.15.45:0/1360124880 learned_addr learned my addr 172.21.15.45:0/1360124880 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 client.?.objecter ms_handle_connect 0x7f41ec13c010 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2741686203 0 0) 0x7f41ec0712e0 con 0x7f41ec13c010 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient(hunting): my global_id is 124169 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41d0004ec0 con 0x7f41ec13c010 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4177984304 0 0) 0x7f41d0004ec0 con 0x7f41ec13c010 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41ec0712e0 con 0x7f41ec13c010 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 client.?.objecter ms_handle_connect 0x7f41ec09fc40 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f41e4001f90 con 0x7f41ec13c010 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient(hunting): dump: 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 548507637 0 0) 0x7f41ec0712e0 con 0x7f41ec13c010 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 >> v1:172.21.15.46:6789/0 conn(0x7f41ec09fc40 legacy=0x7f41ec06e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 >> v1:172.21.15.45:6789/0 conn(0x7f41ec135950 legacy=0x7f41ec06ed20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient: _finish_hunting 0 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 monclient: found mon.c 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:41.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec070580 con 0x7f41ec13c010 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient: _finish_auth 0 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 5 monclient: authenticate success, global_id 124169 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: _renew_subs 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f41ec070070 con 0x7f41ec13c010 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 client.124169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: _renew_subs 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f41ec06f430 con 0x7f41ec13c010 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41f41a8640 1 librados: init done 2024-09-16T03:12:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41e4005000 con 0x7f41ec13c010 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f41e40051b0 con 0x7f41ec13c010 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e3fff640 4 set_mon_vals no callback set 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.196+0000 7f41e27fc640 10 monclient: dump: 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41f41a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41f41a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41f41a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41d4005090 con 0x7f41ec13c010 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f41e4011220 con 0x7f41ec13c010 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(344..344 src has 1..344) ==== 5386+0+0 (unknown 2673046086 0 0) 0x7f41e404bb80 con 0x7f41ec13c010 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41e27fc640 10 client.124169.objecter ms_dispatch 0x7f41ec000ba0 osd_map(344..344 src has 1..344) 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41e27fc640 3 client.124169.objecter handle_osd_map got epochs [344,344] > 0 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41e27fc640 3 client.124169.objecter handle_osd_map decoding full epoch 344 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41e27fc640 20 client.124169.objecter dump_active .. 0 homeless 2024-09-16T03:12:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41f1f44640 10 monclient: get_auth_request con 0x7f41d003ded0 auth_method 0 2024-09-16T03:12:41.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.200+0000 7f41e27fc640 10 client.124169.objecter ms_handle_connect 0x7f41d003ded0 2024-09-16T03:12:41.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.204+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f41e40168d0 con 0x7f41ec13c010 2024-09-16T03:12:41.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.204+0000 7f41e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:41.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.204+0000 7f41e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:41.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.440+0000 7f41f41a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:12:41.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.440+0000 7f41f41a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:41.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.440+0000 7f41f41a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:41.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.440+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f41d4005380 con 0x7f41ec13c010 2024-09-16T03:12:41.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41e27fc640 1 -- 172.21.15.45:0/1360124880 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 677431798) 0x7f41e4049260 con 0x7f41ec13c010 2024-09-16T03:12:41.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:41.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41e27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:41.448 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 10 librados: watch_flush enter 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 10 librados: watch_flush exit 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 20 client.124169.objecter shutdown clearing up homeless session... 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 10 client.124169.objecter successfully canceled tick 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 >> v1:172.21.15.46:6812/149360280 conn(0x7f41d003ded0 legacy=0x7f41d0040390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 10 monclient: shutdown 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 >> v1:172.21.15.45:6790/0 conn(0x7f41ec13c010 legacy=0x7f41ec1d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 shutdown_connections 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 >> 172.21.15.45:0/1360124880 conn(0x7f41ec0a2bd0 msgr2=0x7f41ec13ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 shutdown_connections 2024-09-16T03:12:41.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 1 -- 172.21.15.45:0/1360124880 wait complete. 2024-09-16T03:12:41.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:41.444+0000 7f41f41a8640 1 librados: shutdown 2024-09-16T03:12:44.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:12:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient: get_monmap_and_config 2024-09-16T03:12:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient: build_initial_monmap 2024-09-16T03:12:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient: monmap: 2024-09-16T03:12:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:44.597824+0000 2024-09-16T03:12:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:44.597824+0000 2024-09-16T03:12:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:44.598 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:44.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 1 Processor -- start 2024-09-16T03:12:44.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 1 -- start start 2024-09-16T03:12:44.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient: init 2024-09-16T03:12:44.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:44.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient(hunting): picked mon.noname-c con 0x7f07e8135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient(hunting): picked mon.noname-a con 0x7f07e809fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient(hunting): picked mon.noname-b con 0x7f07e813c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e809f7e0 con 0x7f07e809fc40 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e809b160 con 0x7f07e8135950 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e809b440 con 0x7f07e813c010 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ef437640 10 monclient(hunting): authenticate will time out at 1455.247180s 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ec9d2640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f07e809fc40 legacy=0x7f07e813a8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ed9d4640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f07e813c010 0x7f07e813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38286/0 (socket says 172.21.15.45:38286) 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07ed9d4640 1 -- 172.21.15.45:0/1647905545 learned_addr learned my addr 172.21.15.45:0/1647905545 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07e7fff640 1 -- 172.21.15.45:0/1647905545 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713582398 0 0) 0x7f07e809b440 con 0x7f07e813c010 2024-09-16T03:12:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07e7fff640 10 monclient(hunting): _negotiate 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07e7fff640 10 monclient(hunting): my global_id is 124172 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.592+0000 7f07e7fff640 1 -- 172.21.15.45:0/1647905545 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d8003530 con 0x7f07e813c010 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 1 -- 172.21.15.45:0/1647905545 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 134789552 0 0) 0x7f07d8003530 con 0x7f07e813c010 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 1 -- 172.21.15.45:0/1647905545 >> v1:172.21.15.46:6789/0 conn(0x7f07e8135950 legacy=0x7f07e8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 1 -- 172.21.15.45:0/1647905545 >> v1:172.21.15.45:6789/0 conn(0x7f07e809fc40 legacy=0x7f07e813a8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 1 monclient: found mon.noname-b 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 1 -- 172.21.15.45:0/1647905545 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e8136440 con 0x7f07e813c010 2024-09-16T03:12:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: _finish_auth 0 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 1 -- 172.21.15.45:0/1647905545 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f07e00016a0 con 0x7f07e813c010 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 5 monclient: authenticate success, global_id 124172 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: dump: 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 1 -- 172.21.15.45:0/1647905545 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07e00018c0 con 0x7f07e813c010 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 1 -- 172.21.15.45:0/1647905545 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f07e0001d50 con 0x7f07e813c010 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e7fff640 10 monclient: dump: 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 4 set_mon_vals no callback set 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: shutdown 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- 172.21.15.45:0/1647905545 >> v1:172.21.15.45:6790/0 conn(0x7f07e813c010 legacy=0x7f07e813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- 172.21.15.45:0/1647905545 shutdown_connections 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- 172.21.15.45:0/1647905545 >> 172.21.15.45:0/1647905545 conn(0x7f07e80a2bd0 msgr2=0x7f07e80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- 172.21.15.45:0/1647905545 shutdown_connections 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- 172.21.15.45:0/1647905545 wait complete. 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: build_initial_monmap 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: monmap: 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 librados: starting msgr at 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 librados: starting objecter 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 Processor -- start 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- start start 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 librados: setting wanted keys 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 librados: calling monclient init 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: init 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient(hunting): picked mon.noname-a con 0x7f07e809fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient(hunting): picked mon.noname-c con 0x7f07e8135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient(hunting): picked mon.noname-b con 0x7f07e813c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e806f1d0 con 0x7f07e809fc40 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e8073400 con 0x7f07e813c010 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e8070020 con 0x7f07e8135950 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient(hunting): authenticate will time out at 1455.250393s 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ed1d3640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f07e809fc40 legacy=0x7f07e806e3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ec9d2640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f07e8135950 0x7f07e806eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38294/0 (socket says 172.21.15.45:38294) 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ec9d2640 1 -- 172.21.15.45:0/970238353 learned_addr learned my addr 172.21.15.45:0/970238353 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 client.?.objecter ms_handle_connect 0x7f07e8135950 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204052176 0 0) 0x7f07e8070020 con 0x7f07e8135950 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:12:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): my global_id is 124175 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07cc004ec0 con 0x7f07e8135950 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 client.?.objecter ms_handle_connect 0x7f07e813c010 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 341090289 0 0) 0x7f07cc004ec0 con 0x7f07e8135950 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07e8070020 con 0x7f07e8135950 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f07dc004560 con 0x7f07e8135950 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): dump: 2024-09-16T03:12:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211679041 0 0) 0x7f07e8073400 con 0x7f07e813c010 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient(hunting): my global_id is 22996 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07cc004ec0 con 0x7f07e813c010 2024-09-16T03:12:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2513081617 0 0) 0x7f07e8070020 con 0x7f07e8135950 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 >> v1:172.21.15.46:6789/0 conn(0x7f07e813c010 legacy=0x7f07e81d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 >> v1:172.21.15.45:6789/0 conn(0x7f07e809fc40 legacy=0x7f07e806e3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 monclient: found mon.c 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e80701f0 con 0x7f07e8135950 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient: _finish_auth 0 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 5 monclient: authenticate success, global_id 124175 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: _renew_subs 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07e8073f40 con 0x7f07e8135950 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 client.124175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: _renew_subs 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f07e81d4470 con 0x7f07e8135950 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07ef437640 1 librados: init done 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07dc004780 con 0x7f07e8135950 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f07dc004c10 con 0x7f07e8135950 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e77fe640 4 set_mon_vals no callback set 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.596+0000 7f07e5ffb640 10 monclient: dump: 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:44.619 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07ef437640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07ef437640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07ef437640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07c8005090 con 0x7f07e8135950 2024-09-16T03:12:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f07dc004db0 con 0x7f07e8135950 2024-09-16T03:12:44.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(344..344 src has 1..344) ==== 5386+0+0 (unknown 2673046086 0 0) 0x7f07dc04bb70 con 0x7f07e8135950 2024-09-16T03:12:44.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07e5ffb640 10 client.124175.objecter ms_dispatch 0x7f07e8000ba0 osd_map(344..344 src has 1..344) 2024-09-16T03:12:44.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07e5ffb640 3 client.124175.objecter handle_osd_map got epochs [344,344] > 0 2024-09-16T03:12:44.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07e5ffb640 3 client.124175.objecter handle_osd_map decoding full epoch 344 2024-09-16T03:12:44.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07e5ffb640 20 client.124175.objecter dump_active .. 0 homeless 2024-09-16T03:12:44.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07ed1d3640 10 monclient: get_auth_request con 0x7f07cc03e060 auth_method 0 2024-09-16T03:12:44.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.600+0000 7f07e5ffb640 10 client.124175.objecter ms_handle_connect 0x7f07cc03e060 2024-09-16T03:12:44.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.604+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f07dc016eb0 con 0x7f07e8135950 2024-09-16T03:12:44.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.604+0000 7f07e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:44.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.604+0000 7f07e5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07ef437640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:12:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07ef437640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07ef437640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f07c8005380 con 0x7f07e8135950 2024-09-16T03:12:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07e5ffb640 1 -- 172.21.15.45:0/970238353 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 392330486) 0x7f07dc0167b0 con 0x7f07e8135950 2024-09-16T03:12:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07e5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:44.842 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07ef437640 10 librados: watch_flush enter 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07ef437640 10 librados: watch_flush exit 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07ef437640 20 client.124175.objecter shutdown clearing up homeless session... 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.836+0000 7f07ef437640 10 client.124175.objecter successfully canceled tick 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.840+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 >> v1:172.21.15.46:6812/149360280 conn(0x7f07cc03e060 legacy=0x7f07cc040520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.840+0000 7f07ef437640 10 monclient: shutdown 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.840+0000 7f07ef437640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.840+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 >> v1:172.21.15.45:6790/0 conn(0x7f07e8135950 legacy=0x7f07e806eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.840+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 shutdown_connections 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.840+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 >> 172.21.15.45:0/970238353 conn(0x7f07e80a2bd0 msgr2=0x7f07e813e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.840+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 shutdown_connections 2024-09-16T03:12:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.840+0000 7f07ef437640 1 -- 172.21.15.45:0/970238353 wait complete. 2024-09-16T03:12:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:44.840+0000 7f07ef437640 1 librados: shutdown 2024-09-16T03:12:47.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:12:47.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient: get_monmap_and_config 2024-09-16T03:12:47.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient: build_initial_monmap 2024-09-16T03:12:47.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient: monmap: 2024-09-16T03:12:47.993 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:47.993744+0000 2024-09-16T03:12:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:47.993744+0000 2024-09-16T03:12:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:47.994 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 1 Processor -- start 2024-09-16T03:12:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 1 -- start start 2024-09-16T03:12:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient: init 2024-09-16T03:12:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:12:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient(hunting): picked mon.noname-b con 0x7f418409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient(hunting): picked mon.noname-c con 0x7f418409b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient(hunting): picked mon.noname-a con 0x7f418413b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418409cc20 con 0x7f418413b000 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418409d820 con 0x7f418409b250 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418409b6a0 con 0x7f418409b8e0 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f418a0c0640 10 monclient(hunting): authenticate will time out at 1458.643045s 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f4183fff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f418413d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f41837fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f418409b8e0 0x7f418409bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38308/0 (socket says 172.21.15.45:38308) 2024-09-16T03:12:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:47.988+0000 7f41837fe640 1 -- 172.21.15.45:0/188572066 learned_addr learned my addr 172.21.15.45:0/188572066 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:48.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:48.192+0000 7f4183fff640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f418413d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:48.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:48.596+0000 7f4183fff640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f418413d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:49.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:49.400+0000 7f4183fff640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f418413d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 10 monclient(hunting): tick 2024-09-16T03:12:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 1 monclient(hunting): continuing hunt 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6790/0 conn(0x7f418409b8e0 legacy=0x7f418409bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.46:6789/0 conn(0x7f418409b250 legacy=0x7f41841398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f418413d3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 10 monclient(hunting): picked mon.noname-a con 0x7f418413b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 10 monclient(hunting): picked mon.noname-c con 0x7f4170009070 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 10 monclient(hunting): picked mon.noname-b con 0x7f418409b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41840a21f0 con 0x7f418413b000 2024-09-16T03:12:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418409cc20 con 0x7f4170009070 2024-09-16T03:12:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41837fe640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4170012610 con 0x7f418409b8e0 2024-09-16T03:12:50.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:50.992+0000 7f41817fa640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:51.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:51.196+0000 7f41837fe640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:51.600+0000 7f41837fe640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:52.404+0000 7f41837fe640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:53.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 10 monclient(hunting): tick 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 1 monclient(hunting): continuing hunt 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6790/0 conn(0x7f418409b8e0 legacy=0x7f417000edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.46:6789/0 conn(0x7f4170009070 legacy=0x7f417000b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 10 monclient(hunting): picked mon.noname-b con 0x7f41700128e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 10 monclient(hunting): picked mon.noname-c con 0x7f4170014bd0 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 10 monclient(hunting): picked mon.noname-a con 0x7f418413b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4170013910 con 0x7f418413b000 2024-09-16T03:12:53.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41840a21f0 con 0x7f4170014bd0 2024-09-16T03:12:53.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 1 -- 172.21.15.45:0/188572066 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417000cdd0 con 0x7f41700128e0 2024-09-16T03:12:53.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f41817fa640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:53.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:53.992+0000 7f4183fff640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:54.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:54.196+0000 7f4183fff640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:54.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:54.600+0000 7f4183fff640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:55.404+0000 7f4183fff640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:56.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 -- 172.21.15.45:0/188572066 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1121480545 0 0) 0x7f417000cdd0 con 0x7f41700128e0 2024-09-16T03:12:56.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient(hunting): my global_id is 124178 2024-09-16T03:12:56.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 -- 172.21.15.45:0/188572066 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f416c003530 con 0x7f41700128e0 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 -- 172.21.15.45:0/188572066 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 321042406 0 0) 0x7f416c003530 con 0x7f41700128e0 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.46:6789/0 conn(0x7f4170014bd0 legacy=0x7f4170016fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f4170007930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: _finish_hunting 0 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 monclient: found mon.noname-b 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 -- 172.21.15.45:0/188572066 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417000e050 con 0x7f41700128e0 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: _finish_auth 0 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: _check_auth_tickets 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 -- 172.21.15.45:0/188572066 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4174001660 con 0x7f41700128e0 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f418a0c0640 5 monclient: authenticate success, global_id 124178 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f418a0c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: dump: 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f418a0c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 -- 172.21.15.45:0/188572066 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4174001870 con 0x7f41700128e0 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f418a0c0640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f418a0c0640 4 set_mon_vals no callback set 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 1 -- 172.21.15.45:0/188572066 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4174001c80 con 0x7f41700128e0 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.036+0000 7f41827fc640 10 monclient: dump: 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.046 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 10 monclient: shutdown 2024-09-16T03:12:56.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:56.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 -- 172.21.15.45:0/188572066 >> v1:172.21.15.45:6790/0 conn(0x7f41700128e0 legacy=0x7f4170012d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 -- 172.21.15.45:0/188572066 shutdown_connections 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 -- 172.21.15.45:0/188572066 >> 172.21.15.45:0/188572066 conn(0x7f41840a2bd0 msgr2=0x7f41840a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 -- 172.21.15.45:0/188572066 shutdown_connections 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 -- 172.21.15.45:0/188572066 wait complete. 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 10 monclient: build_initial_monmap 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 10 monclient: monmap: 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 librados: starting msgr at 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 librados: starting objecter 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 Processor -- start 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 -- start start 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 librados: setting wanted keys 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 1 librados: calling monclient init 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.040+0000 7f418a0c0640 10 monclient: init 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 10 monclient(hunting): picked mon.noname-c con 0x7f41700128e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 10 monclient(hunting): picked mon.noname-b con 0x7f4170014bd0 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 10 monclient(hunting): picked mon.noname-a con 0x7f418413b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4184094c10 con 0x7f418413b000 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f41837fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f41700128e0 0x7f418413a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59164/0 (socket says 172.21.15.45:59164) 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f41837fe640 1 -- 172.21.15.45:0/329168742 learned_addr learned my addr 172.21.15.45:0/329168742 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4184094ef0 con 0x7f4170014bd0 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f4183fff640 1 -- 172.21.15.45:0/329168742 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f41841d08d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41840936f0 con 0x7f41700128e0 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f416bfff640 10 client.?.objecter ms_handle_connect 0x7f41700128e0 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f416bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f418a0c0640 10 monclient(hunting): authenticate will time out at 1466.696679s 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f416bfff640 10 client.?.objecter ms_handle_connect 0x7f4170014bd0 2024-09-16T03:12:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.044+0000 7f416bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:56.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.052+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3716344348 0 0) 0x7f41840936f0 con 0x7f41700128e0 2024-09-16T03:12:56.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.052+0000 7f416bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.052+0000 7f416bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.052+0000 7f416bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.052+0000 7f416bfff640 10 monclient(hunting): my global_id is 124181 2024-09-16T03:12:56.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.052+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41700137f0 con 0x7f41700128e0 2024-09-16T03:12:56.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.052+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4032346613 0 0) 0x7f41700137f0 con 0x7f41700128e0 2024-09-16T03:12:56.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.052+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41840936f0 con 0x7f41700128e0 2024-09-16T03:12:56.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.052+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4174001bc0 con 0x7f41700128e0 2024-09-16T03:12:56.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:56.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:56.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient(hunting): dump: 2024-09-16T03:12:56.061 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.061 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2204129596 0 0) 0x7f41840936f0 con 0x7f41700128e0 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 >> v1:172.21.15.46:6789/0 conn(0x7f4170014bd0 legacy=0x7f41841cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 >> v1:172.21.15.45:6789/0 conn(0x7f418413b000 legacy=0x7f41841d08d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:12:56.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 1 monclient: found mon.c 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4184092bd0 con 0x7f41700128e0 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient: _finish_auth 0 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 5 monclient: authenticate success, global_id 124181 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 10 monclient: _renew_subs 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4184093bb0 con 0x7f41700128e0 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 10 client.124181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 10 monclient: _renew_subs 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f41841d1ff0 con 0x7f41700128e0 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 1 librados: init done 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4174001870 con 0x7f41700128e0 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4174004c40 con 0x7f41700128e0 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f4181ffb640 4 set_mon_vals no callback set 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 monclient: dump: 2024-09-16T03:12:56.064 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.065 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.065 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.065 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.065 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.065 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.065 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.065 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.065 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.065 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:56.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4174004de0 con 0x7f41700128e0 2024-09-16T03:12:56.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:56.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4164005090 con 0x7f41700128e0 2024-09-16T03:12:56.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(344..344 src has 1..344) ==== 5386+0+0 (unknown 2673046086 0 0) 0x7f417404b7b0 con 0x7f41700128e0 2024-09-16T03:12:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 client.124181.objecter ms_dispatch 0x7f4184000ba0 osd_map(344..344 src has 1..344) 2024-09-16T03:12:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 3 client.124181.objecter handle_osd_map got epochs [344,344] > 0 2024-09-16T03:12:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 3 client.124181.objecter handle_osd_map decoding full epoch 344 2024-09-16T03:12:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 20 client.124181.objecter dump_active .. 0 homeless 2024-09-16T03:12:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f4182ffd640 10 monclient: get_auth_request con 0x7f4170041610 auth_method 0 2024-09-16T03:12:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.056+0000 7f416bfff640 10 client.124181.objecter ms_handle_connect 0x7f4170041610 2024-09-16T03:12:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.060+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4174016af0 con 0x7f41700128e0 2024-09-16T03:12:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.060+0000 7f416bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.060+0000 7f416bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:56.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.320+0000 7f418a0c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:12:56.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.320+0000 7f418a0c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:56.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.320+0000 7f418a0c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.320+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4164005380 con 0x7f41700128e0 2024-09-16T03:12:56.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.320+0000 7f416bfff640 1 -- 172.21.15.45:0/329168742 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 910906112) 0x7f41740163f0 con 0x7f41700128e0 2024-09-16T03:12:56.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.320+0000 7f416bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:12:56.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.320+0000 7f416bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:56.326 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":36,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 10 librados: watch_flush enter 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 10 librados: watch_flush exit 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 20 client.124181.objecter shutdown clearing up homeless session... 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 10 client.124181.objecter successfully canceled tick 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 >> v1:172.21.15.46:6812/149360280 conn(0x7f4170041610 legacy=0x7f4170043ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 10 monclient: shutdown 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 >> v1:172.21.15.45:6790/0 conn(0x7f41700128e0 legacy=0x7f418413a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 shutdown_connections 2024-09-16T03:12:56.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 >> 172.21.15.45:0/329168742 conn(0x7f41840a2bd0 msgr2=0x7f418409ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 shutdown_connections 2024-09-16T03:12:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 1 -- 172.21.15.45:0/329168742 wait complete. 2024-09-16T03:12:56.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.324+0000 7f418a0c0640 1 librados: shutdown 2024-09-16T03:12:56.346 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:12:56.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:12:56.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:12:56.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient: get_monmap_and_config 2024-09-16T03:12:56.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient: build_initial_monmap 2024-09-16T03:12:56.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient: monmap: 2024-09-16T03:12:56.464 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:56.464 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.465 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:56.464806+0000 2024-09-16T03:12:56.465 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:56.464806+0000 2024-09-16T03:12:56.465 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:56.465 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.465 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:56.465 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:56.465 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:56.466 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 1 Processor -- start 2024-09-16T03:12:56.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 1 -- start start 2024-09-16T03:12:56.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient: init 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient(hunting): picked mon.noname-b con 0x7fd0300a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient(hunting): picked mon.noname-a con 0x7fd0300a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient(hunting): picked mon.noname-c con 0x7fd030073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03009d9e0 con 0x7fd0300a5810 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03009dc20 con 0x7fd030073d40 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd030073360 con 0x7fd0300a4d20 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient(hunting): authenticate will time out at 1467.114667s 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02dd9b640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fd0300a5810 legacy=0x7fd03013cf30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02e59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fd0300a4d20 0x7fd0300a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59168/0 (socket says 172.21.15.45:59168) 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02e59c640 1 -- 172.21.15.45:0/3530558264 learned_addr learned my addr 172.21.15.45:0/3530558264 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 -- 172.21.15.45:0/3530558264 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786799455 0 0) 0x7fd030073360 con 0x7fd0300a4d20 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient(hunting): my global_id is 124184 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 -- 172.21.15.45:0/3530558264 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd018003530 con 0x7fd0300a4d20 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 -- 172.21.15.45:0/3530558264 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3246775282 0 0) 0x7fd018003530 con 0x7fd0300a4d20 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 -- 172.21.15.45:0/3530558264 >> v1:172.21.15.46:6789/0 conn(0x7fd030073d40 legacy=0x7fd030140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 -- 172.21.15.45:0/3530558264 >> v1:172.21.15.45:6789/0 conn(0x7fd0300a5810 legacy=0x7fd03013cf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: _finish_hunting 0 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 monclient: found mon.noname-b 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 -- 172.21.15.45:0/3530558264 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03009cf30 con 0x7fd0300a4d20 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: _finish_auth 0 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: _check_auth_tickets 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 5 monclient: authenticate success, global_id 124184 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 -- 172.21.15.45:0/3530558264 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd0200016a0 con 0x7fd0300a4d20 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: dump: 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.479 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 -- 172.21.15.45:0/3530558264 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd0200018c0 con 0x7fd0300a4d20 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd034ffe640 4 set_mon_vals no callback set 2024-09-16T03:12:56.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 1 -- 172.21.15.45:0/3530558264 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd020001d50 con 0x7fd0300a4d20 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.460+0000 7fd02d59a640 10 monclient: dump: 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: shutdown 2024-09-16T03:12:56.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:56.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- 172.21.15.45:0/3530558264 >> v1:172.21.15.45:6790/0 conn(0x7fd0300a4d20 legacy=0x7fd0300a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- 172.21.15.45:0/3530558264 shutdown_connections 2024-09-16T03:12:56.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- 172.21.15.45:0/3530558264 >> 172.21.15.45:0/3530558264 conn(0x7fd0300aa720 msgr2=0x7fd0300aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:56.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- 172.21.15.45:0/3530558264 shutdown_connections 2024-09-16T03:12:56.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- 172.21.15.45:0/3530558264 wait complete. 2024-09-16T03:12:56.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: build_initial_monmap 2024-09-16T03:12:56.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: monmap: 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 librados: starting msgr at 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 librados: starting objecter 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 Processor -- start 2024-09-16T03:12:56.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- start start 2024-09-16T03:12:56.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 librados: setting wanted keys 2024-09-16T03:12:56.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 librados: calling monclient init 2024-09-16T03:12:56.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: init 2024-09-16T03:12:56.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:56.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:56.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient(hunting): picked mon.noname-b con 0x7fd030073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:56.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient(hunting): picked mon.noname-a con 0x7fd0300a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient(hunting): picked mon.noname-c con 0x7fd0300a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0300a1510 con 0x7fd0300a4d20 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03013fe30 con 0x7fd030073d40 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd02dd9b640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fd0300a4d20 legacy=0x7fd03013cec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd030140110 con 0x7fd0300a5810 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient(hunting): authenticate will time out at 1467.117685s 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd02ed9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fd0300a5810 0x7fd0301d4fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59178/0 (socket says 172.21.15.45:59178) 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd02ed9d640 1 -- 172.21.15.45:0/615036392 learned_addr learned my addr 172.21.15.45:0/615036392 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 client.?.objecter ms_handle_connect 0x7fd0300a5810 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:56.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 892380806 0 0) 0x7fd030140110 con 0x7fd0300a5810 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): my global_id is 124187 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd010004ec0 con 0x7fd0300a5810 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 client.?.objecter ms_handle_connect 0x7fd030073d40 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2192050462 0 0) 0x7fd010004ec0 con 0x7fd0300a5810 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd030140110 con 0x7fd0300a5810 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd028001d40 con 0x7fd0300a5810 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): dump: 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250455739 0 0) 0x7fd03013fe30 con 0x7fd030073d40 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): my global_id is 23002 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd010004ec0 con 0x7fd030073d40 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1668477234 0 0) 0x7fd030140110 con 0x7fd0300a5810 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 >> v1:172.21.15.46:6789/0 conn(0x7fd030073d40 legacy=0x7fd03013c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 >> v1:172.21.15.45:6789/0 conn(0x7fd0300a4d20 legacy=0x7fd03013cec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient: _finish_hunting 0 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 monclient: found mon.c 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03013d5d0 con 0x7fd0300a5810 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient: _finish_auth 0 2024-09-16T03:12:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient: _check_auth_tickets 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 5 monclient: authenticate success, global_id 124187 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: _renew_subs 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd03013ebf0 con 0x7fd0300a5810 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 client.124187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: _renew_subs 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd030138750 con 0x7fd0300a5810 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 librados: init done 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd028002000 con 0x7fd0300a5810 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd02cd99640 4 set_mon_vals no callback set 2024-09-16T03:12:56.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd028003df0 con 0x7fd0300a5810 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient: dump: 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd0140065d0 con 0x7fd014002870 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 client.124187.objecter ms_handle_connect 0x7fd014002870 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 client.124187.objecter resend_mon_ops 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4188624443 0 0) 0x7fd0140065d0 con 0x7fd014002870 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 monclient(hunting): my global_id is 124190 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd030140110 con 0x7fd014002870 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fd028004050 con 0x7fd0300a5810 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(345..345 src has 1..345) ==== 5387+0+0 (unknown 3692503184 0 0) 0x7fd02804baf0 con 0x7fd0300a5810 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 10 client.124187.objecter ms_dispatch 0x7fd030000ba0 osd_map(345..345 src has 1..345) 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 3 client.124187.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:12:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.464+0000 7fd01effd640 3 client.124187.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 20 client.124187.objecter dump_active .. 0 homeless 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4261826484 0 0) 0x7fd030140110 con 0x7fd014002870 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0140065d0 con 0x7fd014002870 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd020001660 con 0x7fd014002870 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 10 monclient: dump: 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.493 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.493 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.493 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.493 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd02dd9b640 10 monclient: get_auth_request con 0x7fd01003e190 auth_method 0 2024-09-16T03:12:56.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4250209287 0 0) 0x7fd0140065d0 con 0x7fd014002870 2024-09-16T03:12:56.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd01400a170 con 0x7fd014002870 2024-09-16T03:12:56.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 10 client.124187.objecter ms_handle_connect 0x7fd01003e190 2024-09-16T03:12:56.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd020001900 con 0x7fd014002870 2024-09-16T03:12:56.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:56.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:56.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 >> v1:172.21.15.45:6790/0 conn(0x7fd014002870 legacy=0x7fd014004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.468+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 >> v1:172.21.15.45:6790/0 conn(0x7fd014002870 legacy=0x7fd014004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:56.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd034ffe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:12:56.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd014009e80 con 0x7fd014009140 2024-09-16T03:12:56.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 client.124187.objecter ms_handle_connect 0x7fd014009140 2024-09-16T03:12:56.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 client.124187.objecter resend_mon_ops 2024-09-16T03:12:56.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 462759248 0 0) 0x7fd014009e80 con 0x7fd014009140 2024-09-16T03:12:56.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 monclient(hunting): my global_id is 124193 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0140065d0 con 0x7fd014009140 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3755828532 0 0) 0x7fd0140065d0 con 0x7fd014009140 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd014009e80 con 0x7fd014009140 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd020004b70 con 0x7fd014009140 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 monclient: dump: 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.496 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3962204858 0 0) 0x7fd014009e80 con 0x7fd014009140 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd01400c420 con 0x7fd014009140 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4123 (unknown 0 0 3145224349) 0x7fd020004b00 con 0x7fd014009140 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:56.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 >> v1:172.21.15.45:6790/0 conn(0x7fd014009140 legacy=0x7fd0140095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.472+0000 7fd01effd640 1 -- 172.21.15.45:0/615036392 >> v1:172.21.15.45:6790/0 conn(0x7fd014009140 legacy=0x7fd0140095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 36, 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 18703, 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:12:56.498 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:56.499 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:56.500 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:56.501 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:56.501 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:56.501 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:56.501 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:12:56.502 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:56.503 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.504 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.505 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:56.506 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:56.509 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:56.510 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:12:56.510 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.510 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.510 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:12:56.510 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:12:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 10 librados: watch_flush enter 2024-09-16T03:12:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 10 librados: watch_flush exit 2024-09-16T03:12:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 20 client.124187.objecter shutdown clearing up homeless session... 2024-09-16T03:12:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 10 client.124187.objecter successfully canceled tick 2024-09-16T03:12:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 >> v1:172.21.15.46:6812/149360280 conn(0x7fd01003e190 legacy=0x7fd010040650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 10 monclient: shutdown 2024-09-16T03:12:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 >> v1:172.21.15.45:6790/0 conn(0x7fd0300a5810 legacy=0x7fd0301d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd02ed9d640 1 -- 172.21.15.45:0/615036392 reap_dead start 2024-09-16T03:12:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd02ed9d640 1 -- 172.21.15.45:0/615036392 reap_dead start 2024-09-16T03:12:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 shutdown_connections 2024-09-16T03:12:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 >> 172.21.15.45:0/615036392 conn(0x7fd0300aa720 msgr2=0x7fd0300a9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 shutdown_connections 2024-09-16T03:12:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 1 -- 172.21.15.45:0/615036392 wait complete. 2024-09-16T03:12:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.476+0000 7fd034ffe640 1 librados: shutdown 2024-09-16T03:12:56.513 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:12:56.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:12:56.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:12:56.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.624+0000 7eff87a22640 10 monclient: get_monmap_and_config 2024-09-16T03:12:56.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.624+0000 7eff87a22640 10 monclient: build_initial_monmap 2024-09-16T03:12:56.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.624+0000 7eff87a22640 10 monclient: monmap: 2024-09-16T03:12:56.631 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:56.631 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.631 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:12:56.631279+0000 2024-09-16T03:12:56.631 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:12:56.631279+0000 2024-09-16T03:12:56.631 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:56.631 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.632 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:56.632 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:12:56.632 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:12:56.632 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.624+0000 7eff87a22640 1 Processor -- start 2024-09-16T03:12:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.624+0000 7eff87a22640 1 -- start start 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.624+0000 7eff87a22640 10 monclient: init 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): picked mon.noname-a con 0x7eff800a25d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): picked mon.noname-b con 0x7eff80137340 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): picked mon.noname-c con 0x7eff8013ae50 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff8009c3f0 con 0x7eff800a25d0 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff8009c6d0 con 0x7eff8013ae50 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff8009c9b0 con 0x7eff80137340 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): authenticate will time out at 1467.280677s 2024-09-16T03:12:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff857be640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7eff800a25d0 legacy=0x7eff800a1ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff84fbd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7eff80137340 0x7eff80139730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59194/0 (socket says 172.21.15.45:59194) 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff84fbd640 1 -- 172.21.15.45:0/670560087 learned_addr learned my addr 172.21.15.45:0/670560087 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 -- 172.21.15.45:0/670560087 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008986592 0 0) 0x7eff8009c9b0 con 0x7eff80137340 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient(hunting): my global_id is 124196 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 -- 172.21.15.45:0/670560087 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff68003530 con 0x7eff80137340 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 -- 172.21.15.45:0/670560087 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 197880434 0 0) 0x7eff68003530 con 0x7eff80137340 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 -- 172.21.15.45:0/670560087 >> v1:172.21.15.46:6789/0 conn(0x7eff8013ae50 legacy=0x7eff8013d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 -- 172.21.15.45:0/670560087 >> v1:172.21.15.45:6789/0 conn(0x7eff800a25d0 legacy=0x7eff800a1ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: _finish_hunting 0 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 monclient: found mon.noname-b 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 -- 172.21.15.45:0/670560087 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8009b350 con 0x7eff80137340 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: _finish_auth 0 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: _check_auth_tickets 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 -- 172.21.15.45:0/670560087 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff7c001a20 con 0x7eff80137340 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: dump: 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 5 monclient: authenticate success, global_id 124196 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 -- 172.21.15.45:0/670560087 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff7c001c40 con 0x7eff80137340 2024-09-16T03:12:56.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient: get_monmap_and_config success 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 4 set_mon_vals no callback set 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 1 -- 172.21.15.45:0/670560087 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff7c002050 con 0x7eff80137340 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff77fff640 10 monclient: dump: 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient: shutdown 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- 172.21.15.45:0/670560087 >> v1:172.21.15.45:6790/0 conn(0x7eff80137340 legacy=0x7eff80139730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- 172.21.15.45:0/670560087 shutdown_connections 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- 172.21.15.45:0/670560087 >> 172.21.15.45:0/670560087 conn(0x7eff800a2bd0 msgr2=0x7eff800a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- 172.21.15.45:0/670560087 shutdown_connections 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- 172.21.15.45:0/670560087 wait complete. 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient: build_initial_monmap 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient: monmap: 2024-09-16T03:12:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 librados: starting msgr at 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 librados: starting objecter 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 Processor -- start 2024-09-16T03:12:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- start start 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 librados: setting wanted keys 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 librados: calling monclient init 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient: init 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient: _reopen_session rank -1 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): picked mon.noname-b con 0x7eff800a25d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): picked mon.noname-a con 0x7eff80137340 addr v1:172.21.15.45:6789/0 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): picked mon.noname-c con 0x7eff8013ae50 addr v1:172.21.15.45:6790/0 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff800a3fd0 con 0x7eff80137340 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff8012f5a0 con 0x7eff800a25d0 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff8009d2c0 con 0x7eff8013ae50 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): _renew_subs 2024-09-16T03:12:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff87a22640 10 monclient(hunting): authenticate will time out at 1467.283781s 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff84fbd640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7eff80137340 legacy=0x7eff8012bf90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff85fbf640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7eff8013ae50 0x7eff8012c6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59202/0 (socket says 172.21.15.45:59202) 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff85fbf640 1 -- 172.21.15.45:0/3300868361 learned_addr learned my addr 172.21.15.45:0/3300868361 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.628+0000 7eff75ffb640 10 client.?.objecter ms_handle_connect 0x7eff8013ae50 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3981763539 0 0) 0x7eff8009d2c0 con 0x7eff8013ae50 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): my global_id is 124199 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff6c004ec0 con 0x7eff8013ae50 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 client.?.objecter ms_handle_connect 0x7eff800a25d0 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:12:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3232817747 0 0) 0x7eff6c004ec0 con 0x7eff8013ae50 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff8009d2c0 con 0x7eff8013ae50 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff700019c0 con 0x7eff8013ae50 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): dump: 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 788163711 0 0) 0x7eff8012f5a0 con 0x7eff800a25d0 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): my global_id is 23008 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff6c004ec0 con 0x7eff800a25d0 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1184147517 0 0) 0x7eff8009d2c0 con 0x7eff8013ae50 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 >> v1:172.21.15.46:6789/0 conn(0x7eff800a25d0 legacy=0x7eff8012b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 >> v1:172.21.15.45:6789/0 conn(0x7eff80137340 legacy=0x7eff8012bf90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 monclient: found mon.c 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8012e580 con 0x7eff8013ae50 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: _finish_auth 0 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 5 monclient: authenticate success, global_id 124199 2024-09-16T03:12:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 10 monclient: _renew_subs 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff8012d240 con 0x7eff8013ae50 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 10 client.124199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 10 monclient: _renew_subs 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7eff8012df70 con 0x7eff8013ae50 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 1 librados: init done 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff70001c80 con 0x7eff8013ae50 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff70003a70 con 0x7eff8013ae50 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff76ffd640 4 set_mon_vals no callback set 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: dump: 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7eff600065d0 con 0x7eff60002870 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7eff70003c90 con 0x7eff8013ae50 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 client.124199.objecter ms_handle_connect 0x7eff60002870 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 client.124199.objecter resend_mon_ops 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(345..345 src has 1..345) ==== 5387+0+0 (unknown 3692503184 0 0) 0x7eff7004b820 con 0x7eff8013ae50 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 client.124199.objecter ms_dispatch 0x7eff80000ba0 osd_map(345..345 src has 1..345) 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 3 client.124199.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 3 client.124199.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 20 client.124199.objecter dump_active .. 0 homeless 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4040659564 0 0) 0x7eff600065d0 con 0x7eff60002870 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient(hunting): my global_id is 23011 2024-09-16T03:12:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff8009d2c0 con 0x7eff60002870 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff84fbd640 10 monclient: get_auth_request con 0x7eff6c03e060 auth_method 0 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 762957062 0 0) 0x7eff8009d2c0 con 0x7eff60002870 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff600065d0 con 0x7eff60002870 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff780040b0 con 0x7eff60002870 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 monclient: dump: 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.656 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 10 client.124199.objecter ms_handle_connect 0x7eff6c03e060 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4033765184 0 0) 0x7eff600065d0 con 0x7eff60002870 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.632+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff6000a170 con 0x7eff60002870 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.636+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7eff78006370 con 0x7eff60002870 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.636+0000 7eff75ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.636+0000 7eff75ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.636+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 >> v1:172.21.15.46:6789/0 conn(0x7eff60002870 legacy=0x7eff60004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.636+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 >> v1:172.21.15.46:6789/0 conn(0x7eff60002870 legacy=0x7eff60004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:56.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff87a22640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:12:56.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7eff60009e60 con 0x7eff60009120 2024-09-16T03:12:56.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 10 client.124199.objecter ms_handle_connect 0x7eff60009120 2024-09-16T03:12:56.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 10 client.124199.objecter resend_mon_ops 2024-09-16T03:12:56.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 246063117 0 0) 0x7eff60009e60 con 0x7eff60009120 2024-09-16T03:12:56.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 10 monclient(hunting): my global_id is 23014 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff600065d0 con 0x7eff60009120 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 790884864 0 0) 0x7eff600065d0 con 0x7eff60009120 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff60009e60 con 0x7eff60009120 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7eff78005b00 con 0x7eff60009120 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:12:56.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.640+0000 7eff75ffb640 10 monclient: dump: 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:12:56.661 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2935627687 0 0) 0x7eff60009e60 con 0x7eff60009120 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7eff6000c400 con 0x7eff60009120 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4293 (unknown 0 0 254427246) 0x7eff78006600 con 0x7eff60009120 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff75ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff75ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 >> v1:172.21.15.46:6789/0 conn(0x7eff60009120 legacy=0x7eff600095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff75ffb640 1 -- 172.21.15.45:0/3300868361 >> v1:172.21.15.46:6789/0 conn(0x7eff60009120 legacy=0x7eff600095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:56.662 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:12:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 36, 2024-09-16T03:12:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 498450, 2024-09-16T03:12:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:12:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:12:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:12:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:12:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:12:56.666 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:56.666 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:56.666 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:56.666 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:56.666 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:56.666 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:56.667 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:56.667 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:56.667 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:56.667 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:56.667 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:56.667 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:12:56.668 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:12:56.669 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:12:56.670 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.671 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:56.672 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.673 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:12:56.674 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.675 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:12:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:12:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:12:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:12:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:12:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:12:56.678 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:12:56.678 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:12:56.678 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:12:56.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:12:56.678 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:12:56.678 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:12:56.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 10 librados: watch_flush enter 2024-09-16T03:12:56.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 10 librados: watch_flush exit 2024-09-16T03:12:56.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 20 client.124199.objecter shutdown clearing up homeless session... 2024-09-16T03:12:56.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 10 client.124199.objecter successfully canceled tick 2024-09-16T03:12:56.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 >> v1:172.21.15.46:6812/149360280 conn(0x7eff6c03e060 legacy=0x7eff6c040520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 10 monclient: shutdown 2024-09-16T03:12:56.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:12:56.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 >> v1:172.21.15.45:6790/0 conn(0x7eff8013ae50 legacy=0x7eff8012c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:12:56.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff85fbf640 1 -- 172.21.15.45:0/3300868361 reap_dead start 2024-09-16T03:12:56.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff85fbf640 1 -- 172.21.15.45:0/3300868361 reap_dead start 2024-09-16T03:12:56.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 shutdown_connections 2024-09-16T03:12:56.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 >> 172.21.15.45:0/3300868361 conn(0x7eff800a2bd0 msgr2=0x7eff800727c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:12:56.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 shutdown_connections 2024-09-16T03:12:56.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 1 -- 172.21.15.45:0/3300868361 wait complete. 2024-09-16T03:12:56.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:12:56.644+0000 7eff87a22640 1 librados: shutdown 2024-09-16T03:12:56.685 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:12:56.685 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:14:16.504 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-09-16T03:14:16.498+0000 7f932961e640 -1 snap_mapper.add_oid found existing snaps mapped on 300:afc31e6d:test-rados-api-smithi046-28678-80::foo:2, removing 2024-09-16T03:14:16.505 INFO:tasks.ceph.osd.4.smithi046.stderr:2024-09-16T03:14:16.498+0000 7f9677869640 -1 snap_mapper.add_oid found existing snaps mapped on 300:afc31e6d:test-rados-api-smithi046-28678-80::foo:2, removing 2024-09-16T03:14:26.686 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T03:14:26.686 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:14:26.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:14:26.691 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:14:26.691 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:14:26.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:14:26.789 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:26.787+0000 7f86a5971e00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 349 -> 0 (sync_force? mon rebuild?) 2024-09-16T03:14:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient: get_monmap_and_config 2024-09-16T03:14:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient: build_initial_monmap 2024-09-16T03:14:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient: monmap: 2024-09-16T03:14:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:26.825984+0000 2024-09-16T03:14:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:26.825984+0000 2024-09-16T03:14:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:26.825 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:26.826 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 1 Processor -- start 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 1 -- start start 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient: init 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient(hunting): picked mon.noname-a con 0x7f71f80a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient(hunting): picked mon.noname-c con 0x7f71f80a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient(hunting): picked mon.noname-b con 0x7f71f8073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f809d9e0 con 0x7f71f80a4d20 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f809dc20 con 0x7f71f80a5810 2024-09-16T03:14:26.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f8073360 con 0x7f71f8073d40 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 10 monclient(hunting): authenticate will time out at 1557.475105s 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fe537640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f71f8073d40 0x7f71f8140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43678/0 (socket says 172.21.15.45:43678) 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fdd36640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f71f80a4d20 0x7f71f80a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45170/0 (socket says 172.21.15.45:45170) 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fe537640 1 -- 172.21.15.45:0/757064559 learned_addr learned my addr 172.21.15.45:0/757064559 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 1 -- 172.21.15.45:0/757064559 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3666777162 0 0) 0x7f71f8073360 con 0x7f71f8073d40 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient(hunting): _negotiate 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient(hunting): my global_id is 124202 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 1 -- 172.21.15.45:0/757064559 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71e4003530 con 0x7f71f8073d40 2024-09-16T03:14:26.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 1 -- 172.21.15.45:0/757064559 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3741969340 0 0) 0x7f71e4003530 con 0x7f71f8073d40 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 1 -- 172.21.15.45:0/757064559 >> v1:172.21.15.46:6789/0 conn(0x7f71f80a5810 legacy=0x7f71f813cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 1 -- 172.21.15.45:0/757064559 >> v1:172.21.15.45:6789/0 conn(0x7f71f80a4d20 legacy=0x7f71f80a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient: _finish_hunting 0 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 1 monclient: found mon.noname-b 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 1 -- 172.21.15.45:0/757064559 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f809cf30 con 0x7f71f8073d40 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient: _finish_auth 0 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient: _check_auth_tickets 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 5 monclient: authenticate success, global_id 124202 2024-09-16T03:14:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 1 -- 172.21.15.45:0/757064559 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f71f00016a0 con 0x7f71f8073d40 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fcd34640 10 monclient: dump: 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:26.839 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.823+0000 7f71fff9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fcd34640 1 -- 172.21.15.45:0/757064559 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71f00018c0 con 0x7f71f8073d40 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fcd34640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fcd34640 1 -- 172.21.15.45:0/757064559 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f71f0001d10 con 0x7f71f8073d40 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fcd34640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fcd34640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fcd34640 10 monclient: dump: 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 4 set_mon_vals no callback set 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: shutdown 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- 172.21.15.45:0/757064559 >> v1:172.21.15.45:6790/0 conn(0x7f71f8073d40 legacy=0x7f71f8140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- 172.21.15.45:0/757064559 shutdown_connections 2024-09-16T03:14:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- 172.21.15.45:0/757064559 >> 172.21.15.45:0/757064559 conn(0x7f71f80aa720 msgr2=0x7f71f80aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- 172.21.15.45:0/757064559 shutdown_connections 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- 172.21.15.45:0/757064559 wait complete. 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: build_initial_monmap 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: monmap: 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 librados: starting msgr at 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 librados: starting objecter 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 Processor -- start 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- start start 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 librados: setting wanted keys 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 librados: calling monclient init 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: init 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient(hunting): picked mon.noname-a con 0x7f71f8073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient(hunting): picked mon.noname-c con 0x7f71f80a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:26.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient(hunting): picked mon.noname-b con 0x7f71f80a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f8140800 con 0x7f71f8073d40 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f813d850 con 0x7f71f80a5810 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f813f990 con 0x7f71f80a4d20 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient(hunting): authenticate will time out at 1557.478254s 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fdd36640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f71f8073d40 0x7f71f813ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45176/0 (socket says 172.21.15.45:45176) 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fdd36640 1 -- 172.21.15.45:0/3632588669 learned_addr learned my addr 172.21.15.45:0/3632588669 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fd535640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f71f80a4d20 0x7f71f813d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43684/0 (socket says 172.21.15.45:43684) 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.?.objecter ms_handle_connect 0x7f71f8073d40 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.?.objecter ms_handle_connect 0x7f71f80a4d20 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1014425577 0 0) 0x7f71f813f990 con 0x7f71f80a4d20 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): my global_id is 124205 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71dc004ec0 con 0x7f71f80a4d20 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.?.objecter ms_handle_connect 0x7f71f80a5810 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2699822377 0 0) 0x7f71dc004ec0 con 0x7f71f80a4d20 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71f813f990 con 0x7f71f80a4d20 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f71e8001900 con 0x7f71f80a4d20 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:14:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): dump: 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3404098332 0 0) 0x7f71f813d850 con 0x7f71f80a5810 2024-09-16T03:14:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): my global_id is 23017 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71dc004ec0 con 0x7f71f80a5810 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3630701596 0 0) 0x7f71f813f990 con 0x7f71f80a4d20 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 >> v1:172.21.15.46:6789/0 conn(0x7f71f80a5810 legacy=0x7f71f81d51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 >> v1:172.21.15.45:6789/0 conn(0x7f71f8073d40 legacy=0x7f71f813ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 monclient: found mon.c 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f813e4a0 con 0x7f71f80a4d20 2024-09-16T03:14:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient: _finish_auth 0 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 5 monclient: authenticate success, global_id 124205 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: _renew_subs 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71f8140520 con 0x7f71f80a4d20 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 client.124205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: _renew_subs 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f71f813f260 con 0x7f71f80a4d20 2024-09-16T03:14:26.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fff9a640 1 librados: init done 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fd535640 1 -- 172.21.15.45:0/3632588669 >> v1:172.21.15.45:6790/0 conn(0x7f71f80a4d20 legacy=0x7f71f813d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fd535640 1 -- 172.21.15.45:0/3632588669 >> v1:172.21.15.45:6790/0 conn(0x7f71f80a4d20 legacy=0x7f71f813d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71e8003620 con 0x7f71f80a4d20 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fd535640 1 --1- 172.21.15.45:0/3632588669 >> v1:172.21.15.45:6790/0 conn(0x7f71f80a4d20 0x7f71f813d140 :-1 s=OPENED pgs=9040 cs=1 l=1).handle_message read tag failed 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71fd535640 1 --1- 172.21.15.45:0/3632588669 >> v1:172.21.15.45:6790/0 conn(0x7f71f80a4d20 0x7f71f813d140 :-1 s=OPENED pgs=9040 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.45:6790/0 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 >> v1:172.21.15.45:6790/0 conn(0x7f71f80a4d20 legacy=0x7f71f813d140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71effff640 4 set_mon_vals no callback set 2024-09-16T03:14:26.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): picked mon.c con 0x7f71dc0061a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): picked mon.b con 0x7f71f80a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): picked mon.a con 0x7f71f8073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71f813f990 con 0x7f71f8073d40 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71f813d850 con 0x7f71f80a5810 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71dc013a10 con 0x7f71dc0061a0 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.124205.objecter ms_handle_connect 0x7f71dc0061a0 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.124205.objecter resend_mon_ops 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.124205.objecter ms_handle_connect 0x7f71f8073d40 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.124205.objecter resend_mon_ops 2024-09-16T03:14:26.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.124205.objecter ms_handle_connect 0x7f71f80a5810 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.827+0000 7f71ee7fc640 10 client.124205.objecter resend_mon_ops 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 822097061 0 0) 0x7f71f813d850 con 0x7f71f80a5810 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.46:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f71dc011910 con 0x7f71f80a5810 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71fff9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71fff9a640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3318532864 0 0) 0x7f71dc013a10 con 0x7f71dc0061a0 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:26.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.45:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f71f813d850 con 0x7f71dc0061a0 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 4284466472 0 0) 0x7f71dc011910 con 0x7f71f80a5810 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 >> v1:172.21.15.45:6790/0 conn(0x7f71dc0061a0 legacy=0x7f71dc0065e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 >> v1:172.21.15.45:6789/0 conn(0x7f71f8073d40 legacy=0x7f71dc00fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 monclient: found mon.b 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f71dc014e30 con 0x7f71f80a5810 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e0005090 con 0x7f71f80a5810 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71dc013a10 con 0x7f71f80a5810 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: _finish_auth 0 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f71e8003c80 con 0x7f71f80a5810 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: dump: 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:26.853 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71e8003e20 con 0x7f71f80a5810 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ef7fe640 4 set_mon_vals no callback set 2024-09-16T03:14:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f71e8004050 con 0x7f71f80a5810 2024-09-16T03:14:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 6 ==== osd_map(387..387 src has 99..387) ==== 5425+0+0 (unknown 3506252399 0 0) 0x7f71e804bb40 con 0x7f71f80a5810 2024-09-16T03:14:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 client.124205.objecter ms_dispatch 0x7f71f8000ba0 osd_map(387..387 src has 99..387) 2024-09-16T03:14:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 3 client.124205.objecter handle_osd_map got epochs [387,387] > 0 2024-09-16T03:14:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 3 client.124205.objecter handle_osd_map decoding full epoch 387 2024-09-16T03:14:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 20 client.124205.objecter dump_active .. 0 homeless 2024-09-16T03:14:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71fdd36640 10 monclient: get_auth_request con 0x7f71dc04a0f0 auth_method 0 2024-09-16T03:14:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.831+0000 7f71ee7fc640 10 client.124205.objecter ms_handle_connect 0x7f71dc04a0f0 2024-09-16T03:14:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.843+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f71e8016f10 con 0x7f71f80a5810 2024-09-16T03:14:26.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.843+0000 7f71ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:26.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.843+0000 7f71ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.867+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f71e8016810 con 0x7f71f80a5810 2024-09-16T03:14:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:26.867+0000 7f71ee7fc640 10 monclient: handle_mon_command_ack 1 not found 2024-09-16T03:14:27.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.099+0000 7f71fff9a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:14:27.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.099+0000 7f71fff9a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:27.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.099+0000 7f71fff9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.46:6789/0 2024-09-16T03:14:27.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.099+0000 7f71fff9a640 1 -- 172.21.15.45:0/3632588669 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f71e0004150 con 0x7f71f80a5810 2024-09-16T03:14:27.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.099+0000 7f71ee7fc640 1 -- 172.21.15.45:0/3632588669 <== mon.1 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 3938841821) 0x7f71e8047110 con 0x7f71f80a5810 2024-09-16T03:14:27.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.099+0000 7f71ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:27.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.099+0000 7f71ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:27.104 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:14:27.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 10 librados: watch_flush enter 2024-09-16T03:14:27.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 10 librados: watch_flush exit 2024-09-16T03:14:27.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 20 client.124205.objecter shutdown clearing up homeless session... 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 10 client.124205.objecter successfully canceled tick 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 1 -- 172.21.15.45:0/3632588669 >> v1:172.21.15.46:6812/149360280 conn(0x7f71dc04a0f0 legacy=0x7f71dc04c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 10 monclient: shutdown 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 1 -- 172.21.15.45:0/3632588669 >> v1:172.21.15.46:6789/0 conn(0x7f71f80a5810 legacy=0x7f71dc00c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 1 -- 172.21.15.45:0/3632588669 shutdown_connections 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 1 -- 172.21.15.45:0/3632588669 >> 172.21.15.45:0/3632588669 conn(0x7f71f80aa720 msgr2=0x7f71f80a72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 1 -- 172.21.15.45:0/3632588669 shutdown_connections 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 1 -- 172.21.15.45:0/3632588669 wait complete. 2024-09-16T03:14:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.103+0000 7f71fff9a640 1 librados: shutdown 2024-09-16T03:14:27.119 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:14:27.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:14:27.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:14:27.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient: get_monmap_and_config 2024-09-16T03:14:27.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient: build_initial_monmap 2024-09-16T03:14:27.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient: monmap: 2024-09-16T03:14:27.240 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:27.240 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.240 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:27.240745+0000 2024-09-16T03:14:27.240 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:27.240745+0000 2024-09-16T03:14:27.240 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:27.240 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.241 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:27.241 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:27.241 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:27.241 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 Processor -- start 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 -- start start 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient: init 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient(hunting): picked mon.noname-b con 0x7f44f409c570 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient(hunting): picked mon.noname-c con 0x7f44f4134320 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient(hunting): picked mon.noname-a con 0x7f44f41347d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f4135150 con 0x7f44f41347d0 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f4074470 con 0x7f44f4134320 2024-09-16T03:14:27.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f4074750 con 0x7f44f409c570 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient(hunting): authenticate will time out at 1557.890065s 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f992b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f44f409c570 0x7f44f409c950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43710/0 (socket says 172.21.15.45:43710) 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fa12c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f44f41347d0 0x7f44f41402a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45200/0 (socket says 172.21.15.45:45200) 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f992b640 1 -- 172.21.15.45:0/3461507145 learned_addr learned my addr 172.21.15.45:0/3461507145 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1646895137 0 0) 0x7f44ec0014a0 con 0x7f44f41347d0 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): my global_id is 134097 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e0003530 con 0x7f44f41347d0 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793339606 0 0) 0x7f44f4074750 con 0x7f44f409c570 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): my global_id is 124208 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44ec0014a0 con 0x7f44f409c570 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1609043147 0 0) 0x7f44f4074470 con 0x7f44f4134320 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient(hunting): my global_id is 23020 2024-09-16T03:14:27.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f4074750 con 0x7f44f4134320 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2834437197 0 0) 0x7f44ec0014a0 con 0x7f44f409c570 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 >> v1:172.21.15.46:6789/0 conn(0x7f44f4134320 legacy=0x7f44f413cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 >> v1:172.21.15.45:6789/0 conn(0x7f44f41347d0 legacy=0x7f44f41402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: _finish_hunting 0 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 monclient: found mon.noname-b 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f4074a30 con 0x7f44f409c570 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: _finish_auth 0 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: _check_auth_tickets 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f44f0001600 con 0x7f44f409c570 2024-09-16T03:14:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:14:27.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: dump: 2024-09-16T03:14:27.248 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.248 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 5 monclient: authenticate success, global_id 124208 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f44f00017a0 con 0x7f44f409c570 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 1 -- 172.21.15.45:0/3461507145 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f44f0001c30 con 0x7f44f409c570 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 4 set_mon_vals no callback set 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44f8929640 10 monclient: dump: 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 10 monclient: shutdown 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 -- 172.21.15.45:0/3461507145 >> v1:172.21.15.45:6790/0 conn(0x7f44f409c570 legacy=0x7f44f409c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 -- 172.21.15.45:0/3461507145 shutdown_connections 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 -- 172.21.15.45:0/3461507145 >> 172.21.15.45:0/3461507145 conn(0x7f44f40a2bd0 msgr2=0x7f44f40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 -- 172.21.15.45:0/3461507145 shutdown_connections 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.239+0000 7f44fbb8f640 1 -- 172.21.15.45:0/3461507145 wait complete. 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient: build_initial_monmap 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient: monmap: 2024-09-16T03:14:27.251 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 librados: starting msgr at 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 librados: starting objecter 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 Processor -- start 2024-09-16T03:14:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 -- start start 2024-09-16T03:14:27.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 librados: setting wanted keys 2024-09-16T03:14:27.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 librados: calling monclient init 2024-09-16T03:14:27.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient: init 2024-09-16T03:14:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient(hunting): picked mon.noname-c con 0x7f44f409c570 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient(hunting): picked mon.noname-a con 0x7f44f4134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient(hunting): picked mon.noname-b con 0x7f44f41347d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f40a5930 con 0x7f44f4134320 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f409d2c0 con 0x7f44f41347d0 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f4133ca0 con 0x7f44f409c570 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient(hunting): authenticate will time out at 1557.893429s 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44f912a640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f44f4134320 0x7f44f4130930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45210/0 (socket says 172.21.15.45:45210) 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44f912a640 1 -- 172.21.15.45:0/524103630 learned_addr learned my addr 172.21.15.45:0/524103630 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 client.?.objecter ms_handle_connect 0x7f44f4134320 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 client.?.objecter ms_handle_connect 0x7f44f409c570 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1203542158 0 0) 0x7f44f40a5930 con 0x7f44f4134320 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): my global_id is 134100 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44d8004ec0 con 0x7f44f4134320 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147941327 0 0) 0x7f44f4133ca0 con 0x7f44f409c570 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): my global_id is 124211 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f40a5930 con 0x7f44f409c570 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 client.?.objecter ms_handle_connect 0x7f44f41347d0 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692502703 0 0) 0x7f44f409d2c0 con 0x7f44f41347d0 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): my global_id is 23023 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f4133ca0 con 0x7f44f41347d0 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3608039075 0 0) 0x7f44f40a5930 con 0x7f44f409c570 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44f409d2c0 con 0x7f44f409c570 2024-09-16T03:14:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 747444777 0 0) 0x7f44d8004ec0 con 0x7f44f4134320 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44f40a5930 con 0x7f44f4134320 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f44f0001b30 con 0x7f44f409c570 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): dump: 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f44e4001940 con 0x7f44f4134320 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient(hunting): dump: 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1119817077 0 0) 0x7f44f409d2c0 con 0x7f44f409c570 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 >> v1:172.21.15.46:6789/0 conn(0x7f44f41347d0 legacy=0x7f44f41d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 >> v1:172.21.15.45:6789/0 conn(0x7f44f4134320 legacy=0x7f44f4130930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 monclient: found mon.c 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f4099990 con 0x7f44f409c570 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient: _finish_auth 0 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 5 monclient: authenticate success, global_id 124211 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient: _renew_subs 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f44f4133400 con 0x7f44f409c570 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 client.124211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient: _renew_subs 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f44f4133040 con 0x7f44f409c570 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 librados: init done 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f44f0004aa0 con 0x7f44f409c570 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f44f0004ef0 con 0x7f44f409c570 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ebfff640 4 set_mon_vals no callback set 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 monclient: dump: 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f44dc0065d0 con 0x7f44dc002870 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 client.124211.objecter ms_handle_connect 0x7f44dc002870 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 client.124211.objecter resend_mon_ops 2024-09-16T03:14:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f44f0005090 con 0x7f44f409c570 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(388..388 src has 99..388) ==== 4973+0+0 (unknown 1220200887 0 0) 0x7f44f004b460 con 0x7f44f409c570 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 10 client.124211.objecter ms_dispatch 0x7f44f4000ba0 osd_map(388..388 src has 99..388) 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 3 client.124211.objecter handle_osd_map got epochs [388,388] > 0 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.243+0000 7f44ea7fc640 3 client.124211.objecter handle_osd_map decoding full epoch 388 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 20 client.124211.objecter dump_active .. 0 homeless 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 967365507 0 0) 0x7f44dc0065d0 con 0x7f44dc002870 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 monclient(hunting): my global_id is 134103 2024-09-16T03:14:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f409d2c0 con 0x7f44dc002870 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44fa12c640 10 monclient: get_auth_request con 0x7f44d803e290 auth_method 0 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2473557639 0 0) 0x7f44f409d2c0 con 0x7f44dc002870 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44dc0065d0 con 0x7f44dc002870 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f44e40037a0 con 0x7f44dc002870 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 monclient: dump: 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 115522257 0 0) 0x7f44dc0065d0 con 0x7f44dc002870 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44dc00a170 con 0x7f44dc002870 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 client.124211.objecter ms_handle_connect 0x7f44d803e290 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f44e40057d0 con 0x7f44dc002870 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 >> v1:172.21.15.45:6789/0 conn(0x7f44dc002870 legacy=0x7f44dc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.247+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 >> v1:172.21.15.45:6789/0 conn(0x7f44dc002870 legacy=0x7f44dc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44fbb8f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f44dc00c500 con 0x7f44dc009120 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 10 client.124211.objecter ms_handle_connect 0x7f44dc009120 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 10 client.124211.objecter resend_mon_ops 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903015623 0 0) 0x7f44dc00c500 con 0x7f44dc009120 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 10 monclient(hunting): my global_id is 134106 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44dc0065d0 con 0x7f44dc009120 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3639286684 0 0) 0x7f44dc0065d0 con 0x7f44dc009120 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44dc00c500 con 0x7f44dc009120 2024-09-16T03:14:27.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f44e4003ce0 con 0x7f44dc009120 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 10 monclient: dump: 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2507553248 0 0) 0x7f44dc00c500 con 0x7f44dc009120 2024-09-16T03:14:27.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.251+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f44dc009cf0 con 0x7f44dc009120 2024-09-16T03:14:27.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3967 (unknown 0 0 3822422041) 0x7f44e4003d00 con 0x7f44dc009120 2024-09-16T03:14:27.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44ea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:14:27.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44ea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:27.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 >> v1:172.21.15.45:6789/0 conn(0x7f44dc009120 legacy=0x7f44dc009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44ea7fc640 1 -- 172.21.15.45:0/524103630 >> v1:172.21.15.45:6789/0 conn(0x7f44dc009120 legacy=0x7f44dc009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:27.272 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:14:27.272 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 40, 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 479, 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:14:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.274 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:14:27.276 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:27.277 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:14:27.278 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:27.279 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:27.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:14:27.281 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:27.282 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:14:27.283 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:14:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 10 librados: watch_flush enter 2024-09-16T03:14:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 10 librados: watch_flush exit 2024-09-16T03:14:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 20 client.124211.objecter shutdown clearing up homeless session... 2024-09-16T03:14:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 10 client.124211.objecter successfully canceled tick 2024-09-16T03:14:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 >> v1:172.21.15.46:6812/149360280 conn(0x7f44d803e290 legacy=0x7f44d8040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 10 monclient: shutdown 2024-09-16T03:14:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 >> v1:172.21.15.45:6790/0 conn(0x7f44f409c570 legacy=0x7f44f4130220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fa12c640 1 -- 172.21.15.45:0/524103630 reap_dead start 2024-09-16T03:14:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fa12c640 1 -- 172.21.15.45:0/524103630 reap_dead start 2024-09-16T03:14:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 shutdown_connections 2024-09-16T03:14:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 >> 172.21.15.45:0/524103630 conn(0x7f44f40a2bd0 msgr2=0x7f44f409f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 shutdown_connections 2024-09-16T03:14:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 1 -- 172.21.15.45:0/524103630 wait complete. 2024-09-16T03:14:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.255+0000 7f44fbb8f640 1 librados: shutdown 2024-09-16T03:14:27.286 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:14:27.287 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:14:27.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:14:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient: get_monmap_and_config 2024-09-16T03:14:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient: build_initial_monmap 2024-09-16T03:14:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient: monmap: 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:27.404594+0000 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:27.404594+0000 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:27.404 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 Processor -- start 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 -- start start 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient: init 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient(hunting): picked mon.noname-a con 0x7f39ac137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient(hunting): picked mon.noname-b con 0x7f39ac138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient(hunting): picked mon.noname-c con 0x7f39ac13e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac0a4f90 con 0x7f39ac137d20 2024-09-16T03:14:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac0a4cb0 con 0x7f39ac13e2a0 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac09dc70 con 0x7f39ac138100 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient(hunting): authenticate will time out at 1558.053941s 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39ab7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f39ac138100 0x7f39ac13cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43718/0 (socket says 172.21.15.45:43718) 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39ab7fe640 1 -- 172.21.15.45:0/307432820 learned_addr learned my addr 172.21.15.45:0/307432820 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1550613757 0 0) 0x7f39ac09dc70 con 0x7f39ac138100 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): my global_id is 124214 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3994003530 con 0x7f39ac138100 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2094568787 0 0) 0x7f39ac0a4f90 con 0x7f39ac137d20 2024-09-16T03:14:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): my global_id is 134109 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39ac09dc70 con 0x7f39ac137d20 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160503863 0 0) 0x7f39ac0a4cb0 con 0x7f39ac13e2a0 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient(hunting): my global_id is 23026 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39ac0a4f90 con 0x7f39ac13e2a0 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2440623370 0 0) 0x7f39ac09dc70 con 0x7f39ac137d20 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 >> v1:172.21.15.45:6790/0 conn(0x7f39ac138100 legacy=0x7f39ac13cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 >> v1:172.21.15.46:6789/0 conn(0x7f39ac13e2a0 legacy=0x7f39ac140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: _finish_hunting 0 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 monclient: found mon.noname-a 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac09cc50 con 0x7f39ac137d20 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: _finish_auth 0 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: _check_auth_tickets 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 5 monclient: authenticate success, global_id 134109 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f39a4001980 con 0x7f39ac137d20 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: dump: 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f39a4001b20 con 0x7f39ac137d20 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 1 -- 172.21.15.45:0/307432820 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f39a4001fb0 con 0x7f39ac137d20 2024-09-16T03:14:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 4 set_mon_vals no callback set 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39aaffd640 10 monclient: dump: 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.413 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 10 monclient: shutdown 2024-09-16T03:14:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 -- 172.21.15.45:0/307432820 >> v1:172.21.15.45:6789/0 conn(0x7f39ac137d20 legacy=0x7f39ac0a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 -- 172.21.15.45:0/307432820 shutdown_connections 2024-09-16T03:14:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 -- 172.21.15.45:0/307432820 >> 172.21.15.45:0/307432820 conn(0x7f39ac0aa110 msgr2=0x7f39ac0aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 -- 172.21.15.45:0/307432820 shutdown_connections 2024-09-16T03:14:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.403+0000 7f39b29b9640 1 -- 172.21.15.45:0/307432820 wait complete. 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient: build_initial_monmap 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient: monmap: 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:27.420 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 librados: starting msgr at 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 librados: starting objecter 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 Processor -- start 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 -- start start 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 librados: setting wanted keys 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 librados: calling monclient init 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient: init 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient(hunting): picked mon.noname-b con 0x7f39ac137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient(hunting): picked mon.noname-a con 0x7f39ac138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient(hunting): picked mon.noname-c con 0x7f39ac13e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac141b10 con 0x7f39ac138100 2024-09-16T03:14:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac0a1300 con 0x7f39ac137d20 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac0a4430 con 0x7f39ac13e2a0 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient(hunting): authenticate will time out at 1558.057865s 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39ab7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f39ac138100 0x7f39ac139030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45242/0 (socket says 172.21.15.45:45242) 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39ab7fe640 1 -- 172.21.15.45:0/3121097834 learned_addr learned my addr 172.21.15.45:0/3121097834 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 client.?.objecter ms_handle_connect 0x7f39ac138100 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 client.?.objecter ms_handle_connect 0x7f39ac13e2a0 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2314770997 0 0) 0x7f39ac141b10 con 0x7f39ac138100 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): my global_id is 134112 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3998004ef0 con 0x7f39ac138100 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 client.?.objecter ms_handle_connect 0x7f39ac137d20 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3724548735 0 0) 0x7f39ac0a4430 con 0x7f39ac13e2a0 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): my global_id is 124217 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39ac141b10 con 0x7f39ac13e2a0 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2237830433 0 0) 0x7f39ac0a1300 con 0x7f39ac137d20 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): my global_id is 23029 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39ac0a4430 con 0x7f39ac137d20 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 432413728 0 0) 0x7f39ac141b10 con 0x7f39ac13e2a0 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39ac0a1300 con 0x7f39ac13e2a0 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f399c0018c0 con 0x7f39ac13e2a0 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): dump: 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 350816853 0 0) 0x7f3998004ef0 con 0x7f39ac138100 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39ac141b10 con 0x7f39ac138100 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f39a00013c0 con 0x7f39ac138100 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): dump: 2024-09-16T03:14:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39ab7fe640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6789/0 conn(0x7f39ac138100 legacy=0x7f39ac139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39ab7fe640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6789/0 conn(0x7f39ac138100 legacy=0x7f39ac139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39ab7fe640 1 --1- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6789/0 conn(0x7f39ac138100 0x7f39ac139030 :-1 s=OPENED pgs=262 cs=1 l=1).handle_message read tag failed 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39ab7fe640 1 --1- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6789/0 conn(0x7f39ac138100 0x7f39ac139030 :-1 s=OPENED pgs=262 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:14:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.45:6789/0 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 591735582 0 0) 0x7f39ac0a1300 con 0x7f39ac13e2a0 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.46:6789/0 conn(0x7f39ac137d20 legacy=0x7f39ac138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6789/0 conn(0x7f39ac138100 legacy=0x7f39ac139030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 monclient: found mon.c 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac13bfb0 con 0x7f39ac13e2a0 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient: _finish_auth 0 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 5 monclient: authenticate success, global_id 124217 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient: _renew_subs 2024-09-16T03:14:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f39ac13b770 con 0x7f39ac13e2a0 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 client.124217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient: _renew_subs 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f39ac13ab50 con 0x7f39ac13e2a0 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 librados: init done 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f399c003620 con 0x7f39ac13e2a0 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f399c003810 con 0x7f39ac13e2a0 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 10 monclient: dump: 2024-09-16T03:14:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39aa7fc640 4 set_mon_vals no callback set 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f398c0065d0 con 0x7f398c002870 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.407+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f399c0039b0 con 0x7f39ac13e2a0 2024-09-16T03:14:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 client.124217.objecter ms_handle_connect 0x7f398c002870 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 client.124217.objecter resend_mon_ops 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(388..388 src has 99..388) ==== 4973+0+0 (unknown 1220200887 0 0) 0x7f399c04a2b0 con 0x7f39ac13e2a0 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 client.124217.objecter ms_dispatch 0x7f39ac000ba0 osd_map(388..388 src has 99..388) 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 3 client.124217.objecter handle_osd_map got epochs [388,388] > 0 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 3 client.124217.objecter handle_osd_map decoding full epoch 388 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 20 client.124217.objecter dump_active .. 0 homeless 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819252983 0 0) 0x7f398c0065d0 con 0x7f398c002870 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 monclient(hunting): my global_id is 124220 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39ac0a1300 con 0x7f398c002870 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39ab7fe640 10 monclient: get_auth_request con 0x7f399803e270 auth_method 0 2024-09-16T03:14:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4140656006 0 0) 0x7f39ac0a1300 con 0x7f398c002870 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f398c0065d0 con 0x7f398c002870 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f39a4001ab0 con 0x7f398c002870 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 monclient: dump: 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 188065812 0 0) 0x7f398c0065d0 con 0x7f398c002870 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f398c00a170 con 0x7f398c002870 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 client.124217.objecter ms_handle_connect 0x7f399803e270 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f39a4001b80 con 0x7f398c002870 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6790/0 conn(0x7f398c002870 legacy=0x7f398c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.411+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6790/0 conn(0x7f398c002870 legacy=0x7f398c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39b29b9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f398c009e80 con 0x7f398c009140 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 10 client.124217.objecter ms_handle_connect 0x7f398c009140 2024-09-16T03:14:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 10 client.124217.objecter resend_mon_ops 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046549654 0 0) 0x7f398c009e80 con 0x7f398c009140 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 10 monclient(hunting): my global_id is 124223 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f398c0065d0 con 0x7f398c009140 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2053325971 0 0) 0x7f398c0065d0 con 0x7f398c009140 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f398c009e80 con 0x7f398c009140 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f39a4003980 con 0x7f398c009140 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.415+0000 7f39a8ff9640 10 monclient: dump: 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2906156710 0 0) 0x7f398c009e80 con 0x7f398c009140 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f398c00c400 con 0x7f398c009140 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 873587156) 0x7f39a4004d00 con 0x7f398c009140 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:14:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:27.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6790/0 conn(0x7f398c009140 legacy=0x7f398c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39a8ff9640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6790/0 conn(0x7f398c009140 legacy=0x7f398c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:27.435 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:14:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:14:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 40, 2024-09-16T03:14:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 109650, 2024-09-16T03:14:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:14:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:14:27.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:27.441 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:27.443 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.444 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.445 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:14:27.446 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:14:27.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 10 librados: watch_flush enter 2024-09-16T03:14:27.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 10 librados: watch_flush exit 2024-09-16T03:14:27.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 20 client.124217.objecter shutdown clearing up homeless session... 2024-09-16T03:14:27.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 10 client.124217.objecter successfully canceled tick 2024-09-16T03:14:27.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.46:6812/149360280 conn(0x7f399803e270 legacy=0x7f3998040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 10 monclient: shutdown 2024-09-16T03:14:27.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:27.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 >> v1:172.21.15.45:6790/0 conn(0x7f39ac13e2a0 legacy=0x7f39ac1d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b0f56640 1 -- 172.21.15.45:0/3121097834 reap_dead start 2024-09-16T03:14:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b0f56640 1 -- 172.21.15.45:0/3121097834 reap_dead start 2024-09-16T03:14:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 shutdown_connections 2024-09-16T03:14:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 >> 172.21.15.45:0/3121097834 conn(0x7f39ac0aa110 msgr2=0x7f39ac0742f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 shutdown_connections 2024-09-16T03:14:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 1 -- 172.21.15.45:0/3121097834 wait complete. 2024-09-16T03:14:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.419+0000 7f39b29b9640 1 librados: shutdown 2024-09-16T03:14:27.449 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:14:27.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:14:27.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:14:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient: get_monmap_and_config 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient: build_initial_monmap 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient: monmap: 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:27.568678+0000 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:27.568678+0000 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:27.568 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 Processor -- start 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 -- start start 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient: init 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient(hunting): picked mon.noname-b con 0x7fd18809c570 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient(hunting): picked mon.noname-c con 0x7fd188134320 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient(hunting): picked mon.noname-a con 0x7fd1881347d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd188135150 con 0x7fd1881347d0 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd188074470 con 0x7fd188134320 2024-09-16T03:14:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd188074750 con 0x7fd18809c570 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient(hunting): authenticate will time out at 1558.218033s 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd186d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd1881347d0 0x7fd1881402a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45258/0 (socket says 172.21.15.45:45258) 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd186d9d640 1 -- 172.21.15.45:0/3078704132 learned_addr learned my addr 172.21.15.45:0/3078704132 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2608282515 0 0) 0x7fd188074750 con 0x7fd18809c570 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): my global_id is 124226 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd168003530 con 0x7fd18809c570 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276248216 0 0) 0x7fd178001120 con 0x7fd1881347d0 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): my global_id is 134115 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd188074750 con 0x7fd1881347d0 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3334364055 0 0) 0x7fd188074470 con 0x7fd188134320 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient(hunting): my global_id is 23032 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd178001120 con 0x7fd188134320 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3395620346 0 0) 0x7fd168003530 con 0x7fd18809c570 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 >> v1:172.21.15.46:6789/0 conn(0x7fd188134320 legacy=0x7fd18813cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 >> v1:172.21.15.45:6789/0 conn(0x7fd1881347d0 legacy=0x7fd1881402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: _finish_hunting 0 2024-09-16T03:14:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 monclient: found mon.noname-b 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd188074a30 con 0x7fd18809c570 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: _finish_auth 0 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: _check_auth_tickets 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd180001a20 con 0x7fd18809c570 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 5 monclient: authenticate success, global_id 124226 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: dump: 2024-09-16T03:14:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd180001bc0 con 0x7fd18809c570 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 1 -- 172.21.15.45:0/3078704132 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd180001f90 con 0x7fd18809c570 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 4 set_mon_vals no callback set 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18559a640 10 monclient: dump: 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 10 monclient: shutdown 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3078704132 >> v1:172.21.15.45:6790/0 conn(0x7fd18809c570 legacy=0x7fd18809c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3078704132 shutdown_connections 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3078704132 >> 172.21.15.45:0/3078704132 conn(0x7fd1880a2bd0 msgr2=0x7fd1880a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3078704132 shutdown_connections 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.567+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3078704132 wait complete. 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient: build_initial_monmap 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient: monmap: 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 librados: starting msgr at 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 librados: starting objecter 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 Processor -- start 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 -- start start 2024-09-16T03:14:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 librados: setting wanted keys 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 librados: calling monclient init 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient: init 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient(hunting): picked mon.noname-c con 0x7fd18809c570 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient(hunting): picked mon.noname-a con 0x7fd188134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient(hunting): picked mon.noname-b con 0x7fd1881347d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd18809d2c0 con 0x7fd188134320 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1880a5870 con 0x7fd1881347d0 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd188133450 con 0x7fd18809c570 2024-09-16T03:14:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient(hunting): authenticate will time out at 1558.221522s 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fd18809c570 0x7fd188130390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43760/0 (socket says 172.21.15.45:43760) 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18659c640 1 -- 172.21.15.45:0/3394353066 learned_addr learned my addr 172.21.15.45:0/3394353066 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 client.?.objecter ms_handle_connect 0x7fd188134320 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 client.?.objecter ms_handle_connect 0x7fd18809c570 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 850450252 0 0) 0x7fd18809d2c0 con 0x7fd188134320 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): my global_id is 134118 2024-09-16T03:14:27.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd16c004ec0 con 0x7fd188134320 2024-09-16T03:14:27.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2578335283 0 0) 0x7fd188133450 con 0x7fd18809c570 2024-09-16T03:14:27.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): my global_id is 124229 2024-09-16T03:14:27.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd18809d2c0 con 0x7fd18809c570 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 client.?.objecter ms_handle_connect 0x7fd1881347d0 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1809233335 0 0) 0x7fd16c004ec0 con 0x7fd188134320 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd188133450 con 0x7fd188134320 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd170001900 con 0x7fd188134320 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): dump: 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.587 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3180230641 0 0) 0x7fd18809d2c0 con 0x7fd18809c570 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd16c004ec0 con 0x7fd18809c570 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd180001f10 con 0x7fd18809c570 2024-09-16T03:14:27.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): dump: 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 807453570 0 0) 0x7fd1880a5870 con 0x7fd1881347d0 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): my global_id is 23035 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd18809d2c0 con 0x7fd1881347d0 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2051294018 0 0) 0x7fd188133450 con 0x7fd188134320 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 >> v1:172.21.15.45:6790/0 conn(0x7fd18809c570 legacy=0x7fd188130390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 >> v1:172.21.15.46:6789/0 conn(0x7fd1881347d0 legacy=0x7fd1881d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient: _finish_hunting 0 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 monclient: found mon.a 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd188133730 con 0x7fd188134320 2024-09-16T03:14:27.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient: _finish_auth 0 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient: _check_auth_tickets 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 5 monclient: authenticate success, global_id 134118 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient: _renew_subs 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1881324f0 con 0x7fd188134320 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 client.134118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient: _renew_subs 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd188131c20 con 0x7fd188134320 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 librados: init done 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd170003620 con 0x7fd188134320 2024-09-16T03:14:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd170003970 con 0x7fd188134320 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd184d99640 4 set_mon_vals no callback set 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient: dump: 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd1600065d0 con 0x7fd160002870 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fd17000f9e0 con 0x7fd188134320 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 client.134118.objecter ms_handle_connect 0x7fd160002870 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 client.134118.objecter resend_mon_ops 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(388..388 src has 99..388) ==== 4973+0+0 (unknown 1220200887 0 0) 0x7fd17004a490 con 0x7fd188134320 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 client.134118.objecter ms_dispatch 0x7fd188000ba0 osd_map(388..388 src has 99..388) 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 3 client.134118.objecter handle_osd_map got epochs [388,388] > 0 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 3 client.134118.objecter handle_osd_map decoding full epoch 388 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 20 client.134118.objecter dump_active .. 0 homeless 2024-09-16T03:14:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895171904 0 0) 0x7fd1600065d0 con 0x7fd160002870 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 10 monclient(hunting): my global_id is 23038 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.571+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd188133450 con 0x7fd160002870 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd186d9d640 10 monclient: get_auth_request con 0x7fd16c03e2b0 auth_method 0 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1858359674 0 0) 0x7fd188133450 con 0x7fd160002870 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1600065d0 con 0x7fd160002870 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 10 client.134118.objecter ms_handle_connect 0x7fd16c03e2b0 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd180003a40 con 0x7fd160002870 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:14:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 10 monclient: dump: 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4064280880 0 0) 0x7fd1600065d0 con 0x7fd160002870 2024-09-16T03:14:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd16000a170 con 0x7fd160002870 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd180005bd0 con 0x7fd160002870 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 >> v1:172.21.15.46:6789/0 conn(0x7fd160002870 legacy=0x7fd160004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.575+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 >> v1:172.21.15.46:6789/0 conn(0x7fd160002870 legacy=0x7fd160004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.579+0000 7fd18cfa0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.579+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd160009e60 con 0x7fd160009120 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 client.134118.objecter ms_handle_connect 0x7fd160009120 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 client.134118.objecter resend_mon_ops 2024-09-16T03:14:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524719018 0 0) 0x7fd160009e60 con 0x7fd160009120 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 monclient(hunting): my global_id is 23041 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1600065d0 con 0x7fd160009120 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 519201464 0 0) 0x7fd1600065d0 con 0x7fd160009120 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd160009e60 con 0x7fd160009120 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd180004030 con 0x7fd160009120 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 monclient: dump: 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3803157234 0 0) 0x7fd160009e60 con 0x7fd160009120 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd16000c3b0 con 0x7fd160009120 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 3444102047) 0x7fd180004070 con 0x7fd160009120 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 >> v1:172.21.15.46:6789/0 conn(0x7fd160009120 legacy=0x7fd1600095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd17effd640 1 -- 172.21.15.45:0/3394353066 >> v1:172.21.15.46:6789/0 conn(0x7fd160009120 legacy=0x7fd1600095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:27.600 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:14:27.600 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:27.600 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:27.600 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:14:27.600 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 40, 2024-09-16T03:14:27.600 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 589385, 2024-09-16T03:14:27.600 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:14:27.600 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:27.601 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:14:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:14:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:14:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.607 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:27.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:14:27.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:27.611 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:27.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:27.612 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:14:27.612 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:14:27.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 10 librados: watch_flush enter 2024-09-16T03:14:27.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 10 librados: watch_flush exit 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 20 client.134118.objecter shutdown clearing up homeless session... 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 10 client.134118.objecter successfully canceled tick 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 >> v1:172.21.15.46:6812/149360280 conn(0x7fd16c03e2b0 legacy=0x7fd16c040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 10 monclient: shutdown 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 >> v1:172.21.15.45:6789/0 conn(0x7fd188134320 legacy=0x7fd188130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd186d9d640 1 -- 172.21.15.45:0/3394353066 reap_dead start 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd186d9d640 1 -- 172.21.15.45:0/3394353066 reap_dead start 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 shutdown_connections 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 >> 172.21.15.45:0/3394353066 conn(0x7fd1880a2bd0 msgr2=0x7fd18809f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 shutdown_connections 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 1 -- 172.21.15.45:0/3394353066 wait complete. 2024-09-16T03:14:27.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.583+0000 7fd18cfa0640 1 librados: shutdown 2024-09-16T03:14:27.615 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:14:27.615 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:14:27.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.731+0000 7fda3ac27640 10 monclient: get_monmap_and_config 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.731+0000 7fda3ac27640 10 monclient: build_initial_monmap 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.731+0000 7fda3ac27640 10 monclient: monmap: 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:27.734900+0000 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:27.734900+0000 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:27.735 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.731+0000 7fda3ac27640 1 Processor -- start 2024-09-16T03:14:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- start start 2024-09-16T03:14:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: init 2024-09-16T03:14:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): picked mon.noname-c con 0x7fda34135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): picked mon.noname-a con 0x7fda3409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): picked mon.noname-b con 0x7fda3413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3409f7e0 con 0x7fda3409fc40 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3409b160 con 0x7fda34135950 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3409b440 con 0x7fda3413c010 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): authenticate will time out at 1558.384748s 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda391c4640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fda3413c010 0x7fda3413e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43774/0 (socket says 172.21.15.45:43774) 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda391c4640 1 -- 172.21.15.45:0/1858781415 learned_addr learned my addr 172.21.15.45:0/1858781415 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda33fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fda3409fc40 0x7fda3413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45270/0 (socket says 172.21.15.45:45270) 2024-09-16T03:14:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1519794869 0 0) 0x7fda3409b440 con 0x7fda3413c010 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): my global_id is 124232 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda24003530 con 0x7fda3413c010 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 681458329 0 0) 0x7fda3409f7e0 con 0x7fda3409fc40 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): my global_id is 134121 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda3409b440 con 0x7fda3409fc40 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3726842970 0 0) 0x7fda3409b160 con 0x7fda34135950 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient(hunting): my global_id is 23044 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda3409f7e0 con 0x7fda34135950 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 548218338 0 0) 0x7fda24003530 con 0x7fda3413c010 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 >> v1:172.21.15.46:6789/0 conn(0x7fda34135950 legacy=0x7fda34135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 >> v1:172.21.15.45:6789/0 conn(0x7fda3409fc40 legacy=0x7fda3413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: _finish_hunting 0 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 monclient: found mon.noname-b 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda34136440 con 0x7fda3413c010 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: _finish_auth 0 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: _check_auth_tickets 2024-09-16T03:14:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda2c001950 con 0x7fda3413c010 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: dump: 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.741 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 5 monclient: authenticate success, global_id 124232 2024-09-16T03:14:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda2c001af0 con 0x7fda3413c010 2024-09-16T03:14:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 1 -- 172.21.15.45:0/1858781415 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda2c001f80 con 0x7fda3413c010 2024-09-16T03:14:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda337fe640 10 monclient: dump: 2024-09-16T03:14:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.745 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.745 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.745 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:27.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 4 set_mon_vals no callback set 2024-09-16T03:14:27.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: shutdown 2024-09-16T03:14:27.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:27.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- 172.21.15.45:0/1858781415 >> v1:172.21.15.45:6790/0 conn(0x7fda3413c010 legacy=0x7fda3413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- 172.21.15.45:0/1858781415 shutdown_connections 2024-09-16T03:14:27.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- 172.21.15.45:0/1858781415 >> 172.21.15.45:0/1858781415 conn(0x7fda340a2bd0 msgr2=0x7fda340a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:27.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- 172.21.15.45:0/1858781415 shutdown_connections 2024-09-16T03:14:27.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- 172.21.15.45:0/1858781415 wait complete. 2024-09-16T03:14:27.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: build_initial_monmap 2024-09-16T03:14:27.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: monmap: 2024-09-16T03:14:27.750 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:27.750 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:27.750 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 librados: starting msgr at 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 librados: starting objecter 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 Processor -- start 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- start start 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 librados: setting wanted keys 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 librados: calling monclient init 2024-09-16T03:14:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: init 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): picked mon.noname-a con 0x7fda3409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): picked mon.noname-b con 0x7fda34135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): picked mon.noname-c con 0x7fda3413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3409d2c0 con 0x7fda3409fc40 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda340740e0 con 0x7fda34135950 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda34071d80 con 0x7fda3413c010 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.735+0000 7fda3ac27640 10 monclient(hunting): authenticate will time out at 1558.387958s 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda389c3640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fda3409fc40 0x7fda34134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45282/0 (socket says 172.21.15.45:45282) 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda389c3640 1 -- 172.21.15.45:0/3087185644 learned_addr learned my addr 172.21.15.45:0/3087185644 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:27.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 client.?.objecter ms_handle_connect 0x7fda3409fc40 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 client.?.objecter ms_handle_connect 0x7fda3413c010 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 735286640 0 0) 0x7fda34071d80 con 0x7fda3413c010 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): my global_id is 124235 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda18004ec0 con 0x7fda3413c010 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515405218 0 0) 0x7fda3409f7e0 con 0x7fda3409fc40 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): my global_id is 134124 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda34071d80 con 0x7fda3409fc40 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 client.?.objecter ms_handle_connect 0x7fda34135950 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda389c3640 1 -- 172.21.15.45:0/3087185644 >> v1:172.21.15.45:6789/0 conn(0x7fda3409fc40 legacy=0x7fda34134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda389c3640 1 -- 172.21.15.45:0/3087185644 >> v1:172.21.15.45:6789/0 conn(0x7fda3409fc40 legacy=0x7fda34134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda389c3640 1 --1- 172.21.15.45:0/3087185644 >> v1:172.21.15.45:6789/0 conn(0x7fda3409fc40 0x7fda34134580 :-1 s=OPENED pgs=266 cs=1 l=1).handle_message read tag failed 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda389c3640 1 --1- 172.21.15.45:0/3087185644 >> v1:172.21.15.45:6789/0 conn(0x7fda3409fc40 0x7fda34134580 :-1 s=OPENED pgs=266 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.45:6789/0 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 960651803 0 0) 0x7fda18004ec0 con 0x7fda3413c010 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda3409f7e0 con 0x7fda3413c010 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda2c001e40 con 0x7fda3413c010 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:27.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): dump: 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1504317076 0 0) 0x7fda340740e0 con 0x7fda34135950 2024-09-16T03:14:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _negotiate 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient(hunting): my global_id is 23047 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda18004ec0 con 0x7fda34135950 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3731534350 0 0) 0x7fda3409f7e0 con 0x7fda3413c010 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 >> v1:172.21.15.46:6789/0 conn(0x7fda34135950 legacy=0x7fda341d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 >> v1:172.21.15.45:6789/0 conn(0x7fda3409fc40 legacy=0x7fda34134580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient: _finish_hunting 0 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 monclient: found mon.c 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda34073e00 con 0x7fda3413c010 2024-09-16T03:14:27.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient: _finish_auth 0 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient: _check_auth_tickets 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 5 monclient: authenticate success, global_id 124235 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 10 monclient: _renew_subs 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda34134c90 con 0x7fda3413c010 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 10 client.124235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 10 monclient: _renew_subs 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fda340712d0 con 0x7fda3413c010 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 1 librados: init done 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda2c001a50 con 0x7fda3413c010 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:27.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fda2c004f80 con 0x7fda3413c010 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 monclient: dump: 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:27.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda32ffd640 4 set_mon_vals no callback set 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda14005090 con 0x7fda3413c010 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fda2c010ff0 con 0x7fda3413c010 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(388..388 src has 99..388) ==== 4973+0+0 (unknown 1220200887 0 0) 0x7fda2c04b670 con 0x7fda3413c010 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 client.124235.objecter ms_dispatch 0x7fda34000ba0 osd_map(388..388 src has 99..388) 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 3 client.124235.objecter handle_osd_map got epochs [388,388] > 0 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 3 client.124235.objecter handle_osd_map decoding full epoch 388 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 20 client.124235.objecter dump_active .. 0 homeless 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda389c3640 10 monclient: get_auth_request con 0x7fda1803e2c0 auth_method 0 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.739+0000 7fda317fa640 10 client.124235.objecter ms_handle_connect 0x7fda1803e2c0 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.743+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fda2c016560 con 0x7fda3413c010 2024-09-16T03:14:27.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.743+0000 7fda317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:27.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.743+0000 7fda317fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:27.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.979+0000 7fda3ac27640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:14:27.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.979+0000 7fda3ac27640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:14:27.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.979+0000 7fda3ac27640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:27.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.979+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fda14005c50 con 0x7fda3413c010 2024-09-16T03:14:27.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.984 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=12,logm=54} crc {auth=1015921045,config=1097430493,health=305190204,kv=4063681046,logm=1055930836}) 2024-09-16T03:14:27.984 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=11,kv=13,logm=54} crc {auth=922721167,config=1097430493,health=3928057762,kv=2649772261,logm=920912528}) 2024-09-16T03:14:27.984 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.984 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=12,logm=54} crc {auth=1015921045,config=1097430493,health=305190204,kv=4063681046,logm=1055930836}) 2024-09-16T03:14:27.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda317fa640 1 -- 172.21.15.45:0/3087185644 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fda2c048ef0 con 0x7fda3413c010 2024-09-16T03:14:27.984 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=12,kv=14,logm=53} crc {auth=2306588168,config=1097430493,health=3623406937,kv=3724913605,logm=1967461228}) 2024-09-16T03:14:27.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:14:27.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda317fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:27.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 10 librados: watch_flush enter 2024-09-16T03:14:27.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 10 librados: watch_flush exit 2024-09-16T03:14:27.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 20 client.124235.objecter shutdown clearing up homeless session... 2024-09-16T03:14:27.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 10 client.124235.objecter successfully canceled tick 2024-09-16T03:14:27.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940706399}) 2024-09-16T03:14:27.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556201502}) 2024-09-16T03:14:27.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 >> v1:172.21.15.46:6812/149360280 conn(0x7fda1803e2c0 legacy=0x7fda18040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 10 monclient: shutdown 2024-09-16T03:14:27.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:27.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 >> v1:172.21.15.45:6790/0 conn(0x7fda3413c010 legacy=0x7fda341d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:27.988 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940706399}) 2024-09-16T03:14:27.988 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.983+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693863643}) 2024-09-16T03:14:27.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 shutdown_connections 2024-09-16T03:14:27.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 >> 172.21.15.45:0/3087185644 conn(0x7fda340a2bd0 msgr2=0x7fda340a1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:27.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 shutdown_connections 2024-09-16T03:14:27.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 1 -- 172.21.15.45:0/3087185644 wait complete. 2024-09-16T03:14:27.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:27.983+0000 7fda3ac27640 1 librados: shutdown 2024-09-16T03:14:27.989 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.989 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940166133}) 2024-09-16T03:14:27.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3032676244}) 2024-09-16T03:14:27.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940166133}) 2024-09-16T03:14:27.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018905777}) 2024-09-16T03:14:27.991 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.991 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247392652}) 2024-09-16T03:14:27.991 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=247457903}) 2024-09-16T03:14:27.991 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.991 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247392652}) 2024-09-16T03:14:27.991 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.987+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905130889}) 2024-09-16T03:14:27.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335752333}) 2024-09-16T03:14:27.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=590743206}) 2024-09-16T03:14:27.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335752333}) 2024-09-16T03:14:27.993 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693839414}) 2024-09-16T03:14:27.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1915036262}) 2024-09-16T03:14:27.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583111994}) 2024-09-16T03:14:27.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1915036262}) 2024-09-16T03:14:27.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.991+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1484294907}) 2024-09-16T03:14:27.995 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.995 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511932300}) 2024-09-16T03:14:27.995 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2715654263}) 2024-09-16T03:14:27.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511932300}) 2024-09-16T03:14:27.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2332140740}) 2024-09-16T03:14:27.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1617408009}) 2024-09-16T03:14:27.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294088441}) 2024-09-16T03:14:27.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1617408009}) 2024-09-16T03:14:27.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402926435}) 2024-09-16T03:14:27.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019289106}) 2024-09-16T03:14:27.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260933837}) 2024-09-16T03:14:27.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:27.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019289106}) 2024-09-16T03:14:27.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.995+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310809685}) 2024-09-16T03:14:28.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875267371}) 2024-09-16T03:14:28.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=483132374}) 2024-09-16T03:14:28.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875267371}) 2024-09-16T03:14:28.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252560446}) 2024-09-16T03:14:28.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534622397}) 2024-09-16T03:14:28.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639688050}) 2024-09-16T03:14:28.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534622397}) 2024-09-16T03:14:28.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:27.999+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973201310}) 2024-09-16T03:14:28.003 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.003 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705054420}) 2024-09-16T03:14:28.003 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=420621381}) 2024-09-16T03:14:28.003 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.003 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705054420}) 2024-09-16T03:14:28.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2793509115}) 2024-09-16T03:14:28.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32414401}) 2024-09-16T03:14:28.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=334467906}) 2024-09-16T03:14:28.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32414401}) 2024-09-16T03:14:28.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2605472025}) 2024-09-16T03:14:28.006 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:14:28.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201186357}) 2024-09-16T03:14:28.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284477492}) 2024-09-16T03:14:28.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201186357}) 2024-09-16T03:14:28.007 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.003+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022728361}) 2024-09-16T03:14:28.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883803900}) 2024-09-16T03:14:28.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2998475027}) 2024-09-16T03:14:28.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883803900}) 2024-09-16T03:14:28.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1776845103}) 2024-09-16T03:14:28.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046065425}) 2024-09-16T03:14:28.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1543161601}) 2024-09-16T03:14:28.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046065425}) 2024-09-16T03:14:28.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.007+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481916445}) 2024-09-16T03:14:28.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2555455384}) 2024-09-16T03:14:28.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3024233254}) 2024-09-16T03:14:28.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2555455384}) 2024-09-16T03:14:28.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4064380717}) 2024-09-16T03:14:28.013 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.013 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001682752}) 2024-09-16T03:14:28.013 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3491476517}) 2024-09-16T03:14:28.013 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.014 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001682752}) 2024-09-16T03:14:28.014 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612165594}) 2024-09-16T03:14:28.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911867513}) 2024-09-16T03:14:28.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110839619}) 2024-09-16T03:14:28.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911867513}) 2024-09-16T03:14:28.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.011+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=354256260}) 2024-09-16T03:14:28.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964558010}) 2024-09-16T03:14:28.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980030649}) 2024-09-16T03:14:28.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964558010}) 2024-09-16T03:14:28.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638924560}) 2024-09-16T03:14:28.018 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.018 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329237045}) 2024-09-16T03:14:28.018 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1362236818}) 2024-09-16T03:14:28.018 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.018 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329237045}) 2024-09-16T03:14:28.018 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.015+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3523810355}) 2024-09-16T03:14:28.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699356157}) 2024-09-16T03:14:28.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3558312027}) 2024-09-16T03:14:28.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699356157}) 2024-09-16T03:14:28.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605797040}) 2024-09-16T03:14:28.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989856251}) 2024-09-16T03:14:28.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2881373061}) 2024-09-16T03:14:28.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989856251}) 2024-09-16T03:14:28.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.019+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=120344634}) 2024-09-16T03:14:28.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124360537}) 2024-09-16T03:14:28.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190178607}) 2024-09-16T03:14:28.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124360537}) 2024-09-16T03:14:28.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168577911}) 2024-09-16T03:14:28.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621553070}) 2024-09-16T03:14:28.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3886740316}) 2024-09-16T03:14:28.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621553070}) 2024-09-16T03:14:28.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964254671}) 2024-09-16T03:14:28.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4122220725}) 2024-09-16T03:14:28.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=635507603}) 2024-09-16T03:14:28.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4122220725}) 2024-09-16T03:14:28.027 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.023+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2767377803}) 2024-09-16T03:14:28.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3080146031}) 2024-09-16T03:14:28.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3695350509}) 2024-09-16T03:14:28.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3080146031}) 2024-09-16T03:14:28.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575070065}) 2024-09-16T03:14:28.029 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.030 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308391500}) 2024-09-16T03:14:28.030 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4254804208}) 2024-09-16T03:14:28.030 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.030 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308391500}) 2024-09-16T03:14:28.030 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.027+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79273806}) 2024-09-16T03:14:28.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663628296}) 2024-09-16T03:14:28.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1495066313}) 2024-09-16T03:14:28.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663628296}) 2024-09-16T03:14:28.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1344538302}) 2024-09-16T03:14:28.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938559527}) 2024-09-16T03:14:28.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2780706594}) 2024-09-16T03:14:28.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938559527}) 2024-09-16T03:14:28.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1756977762}) 2024-09-16T03:14:28.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178469786}) 2024-09-16T03:14:28.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1449653698}) 2024-09-16T03:14:28.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178469786}) 2024-09-16T03:14:28.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.031+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=432003306}) 2024-09-16T03:14:28.036 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.036 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810751075}) 2024-09-16T03:14:28.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363345894}) 2024-09-16T03:14:28.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810751075}) 2024-09-16T03:14:28.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927951673}) 2024-09-16T03:14:28.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237034446}) 2024-09-16T03:14:28.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2456285754}) 2024-09-16T03:14:28.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237034446}) 2024-09-16T03:14:28.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.035+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273538759}) 2024-09-16T03:14:28.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436777853}) 2024-09-16T03:14:28.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=364192108}) 2024-09-16T03:14:28.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436777853}) 2024-09-16T03:14:28.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3866612479}) 2024-09-16T03:14:28.041 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.041 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685591908}) 2024-09-16T03:14:28.041 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=701545700}) 2024-09-16T03:14:28.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685591908}) 2024-09-16T03:14:28.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.039+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055226007}) 2024-09-16T03:14:28.043 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.043 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=691136916}) 2024-09-16T03:14:28.043 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1009488562}) 2024-09-16T03:14:28.043 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.044 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=691136916}) 2024-09-16T03:14:28.044 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=217023151}) 2024-09-16T03:14:28.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758056525}) 2024-09-16T03:14:28.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2362061231}) 2024-09-16T03:14:28.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758056525}) 2024-09-16T03:14:28.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996975517}) 2024-09-16T03:14:28.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486061866}) 2024-09-16T03:14:28.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757940692}) 2024-09-16T03:14:28.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486061866}) 2024-09-16T03:14:28.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.043+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=52101701}) 2024-09-16T03:14:28.048 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.048 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237555930}) 2024-09-16T03:14:28.048 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4221146222}) 2024-09-16T03:14:28.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237555930}) 2024-09-16T03:14:28.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=848702044}) 2024-09-16T03:14:28.050 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.050 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230684287}) 2024-09-16T03:14:28.050 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060821156}) 2024-09-16T03:14:28.050 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.050 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230684287}) 2024-09-16T03:14:28.050 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.047+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=40008587}) 2024-09-16T03:14:28.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389351650}) 2024-09-16T03:14:28.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1654811919}) 2024-09-16T03:14:28.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389351650}) 2024-09-16T03:14:28.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=896076700}) 2024-09-16T03:14:28.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3793242740}) 2024-09-16T03:14:28.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3735698772}) 2024-09-16T03:14:28.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.054 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3793242740}) 2024-09-16T03:14:28.054 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.051+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966438675}) 2024-09-16T03:14:28.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=3320410579,mdsmap=2287870078,mgr=2199327819,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=736242096,mon_config_key=1169163791,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2004720308,osd_pg_creating=3550936167,osd_snap=4074265486}) 2024-09-16T03:14:28.056 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=7} crc {logm=1165576776,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgrstat=4279801759,mon_config_key=2478096643,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2004720307,osd_snap=1067233243}) 2024-09-16T03:14:28.056 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.056 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=3320410579,mdsmap=2287870078,mgr=2199327819,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=736242096,mon_config_key=1169163791,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2004720308,osd_pg_creating=3550936167,osd_snap=4074265486}) 2024-09-16T03:14:28.056 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=44,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=4} crc {logm=1195875004,mdsmap=3244449113,mgr=3993191508,mgr_metadata=127159686,mgrstat=3164062716,mon_config_key=1025616746,monmap=3372108098,nvmeofgw=1596038039,osd_metadata=2004720307,osd_pg_creating=3550936167,osd_snap=3084015885}) 2024-09-16T03:14:28.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=2388476282,osdmap=1226928654}) 2024-09-16T03:14:28.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=3455198206,osdmap=3110256100}) 2024-09-16T03:14:28.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=2388476282,osdmap=1226928654}) 2024-09-16T03:14:28.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.055+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=22,osdmap=78} crc {osd_snap=3819311384,osdmap=3147680711}) 2024-09-16T03:14:28.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4258185453}) 2024-09-16T03:14:28.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1433264407}) 2024-09-16T03:14:28.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4258185453}) 2024-09-16T03:14:28.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2200792109}) 2024-09-16T03:14:28.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2240719699}) 2024-09-16T03:14:28.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2646805588}) 2024-09-16T03:14:28.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2240719699}) 2024-09-16T03:14:28.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.063+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2345234444}) 2024-09-16T03:14:28.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.067+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.067+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3003019022}) 2024-09-16T03:14:28.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.067+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4276788013}) 2024-09-16T03:14:28.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.067+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.067+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3003019022}) 2024-09-16T03:14:28.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.067+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1438166292}) 2024-09-16T03:14:28.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.071+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.071+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=728997900}) 2024-09-16T03:14:28.074 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.071+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=1901432910}) 2024-09-16T03:14:28.074 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.071+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:14:28.074 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.071+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=728997900}) 2024-09-16T03:14:28.074 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:14:28.071+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=54} crc {osdmap=1753597048}) 2024-09-16T03:14:29.007 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:14:29.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:14:29.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient: get_monmap_and_config 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient: build_initial_monmap 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient: monmap: 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:29.156731+0000 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:29.156731+0000 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:29.156 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 1 Processor -- start 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 1 -- start start 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient: init 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient(hunting): picked mon.noname-b con 0x7fe6b0135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient(hunting): picked mon.noname-c con 0x7fe6b009fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient(hunting): picked mon.noname-a con 0x7fe6b013c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b009f7e0 con 0x7fe6b013c010 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b009b160 con 0x7fe6b009fc40 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b009b440 con 0x7fe6b0135950 2024-09-16T03:14:29.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient(hunting): authenticate will time out at 1559.806585s 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6affff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe6b0135950 0x7fe6b0135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43792/0 (socket says 172.21.15.45:43792) 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6affff640 1 -- 172.21.15.45:0/2075361507 learned_addr learned my addr 172.21.15.45:0/2075361507 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b4b7c640 1 --1- 172.21.15.45:0/2075361507 >> v1:172.21.15.45:6789/0 conn(0x7fe6b013c010 0x7fe6b013e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45288/0 (socket says 172.21.15.45:45288) 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050184585 0 0) 0x7fe6b009b440 con 0x7fe6b0135950 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient(hunting): my global_id is 124238 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe698003530 con 0x7fe6b0135950 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193949919 0 0) 0x7fe6b009f7e0 con 0x7fe6b013c010 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient(hunting): my global_id is 134127 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6b009b440 con 0x7fe6b013c010 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1331191314 0 0) 0x7fe698003530 con 0x7fe6b0135950 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 >> v1:172.21.15.46:6789/0 conn(0x7fe6b009fc40 legacy=0x7fe6b013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 >> v1:172.21.15.45:6789/0 conn(0x7fe6b013c010 legacy=0x7fe6b013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: _finish_hunting 0 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 monclient: found mon.noname-b 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b0136440 con 0x7fe6b0135950 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: _finish_auth 0 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 5 monclient: authenticate success, global_id 124238 2024-09-16T03:14:29.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe6a00015d0 con 0x7fe6b0135950 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: dump: 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.167 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6a0001770 con 0x7fe6b0135950 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 4 set_mon_vals no callback set 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 1 -- 172.21.15.45:0/2075361507 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe6a0001c00 con 0x7fe6b0135950 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6aeffd640 10 monclient: dump: 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 10 monclient: shutdown 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.155+0000 7fe6b65df640 1 -- 172.21.15.45:0/2075361507 >> v1:172.21.15.45:6790/0 conn(0x7fe6b0135950 legacy=0x7fe6b0135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- 172.21.15.45:0/2075361507 shutdown_connections 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- 172.21.15.45:0/2075361507 >> 172.21.15.45:0/2075361507 conn(0x7fe6b00a2bd0 msgr2=0x7fe6b00a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- 172.21.15.45:0/2075361507 shutdown_connections 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- 172.21.15.45:0/2075361507 wait complete. 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: build_initial_monmap 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: monmap: 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 librados: starting msgr at 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 librados: starting objecter 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 Processor -- start 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- start start 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 librados: setting wanted keys 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 librados: calling monclient init 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: init 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient(hunting): picked mon.noname-c con 0x7fe6b009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient(hunting): picked mon.noname-b con 0x7fe6b013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient(hunting): picked mon.noname-a con 0x7fe6b0070e40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b0073a60 con 0x7fe6b0070e40 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b013abd0 con 0x7fe6b013c010 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b0073780 con 0x7fe6b009fc40 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient(hunting): authenticate will time out at 1559.809833s 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6affff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe6b009fc40 0x7fe6b0070020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43806/0 (socket says 172.21.15.45:43806) 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6affff640 1 -- 172.21.15.45:0/3938108938 learned_addr learned my addr 172.21.15.45:0/3938108938 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 client.?.objecter ms_handle_connect 0x7fe6b0070e40 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 client.?.objecter ms_handle_connect 0x7fe6b009fc40 2024-09-16T03:14:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213155323 0 0) 0x7fe6b0073a60 con 0x7fe6b0070e40 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): my global_id is 134130 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe69c004ec0 con 0x7fe6b0070e40 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2491422066 0 0) 0x7fe6b0073780 con 0x7fe6b009fc40 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): my global_id is 124241 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6b0073a60 con 0x7fe6b009fc40 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 client.?.objecter ms_handle_connect 0x7fe6b013c010 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3676351538 0 0) 0x7fe69c004ec0 con 0x7fe6b0070e40 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6b0073780 con 0x7fe6b0070e40 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe6a80014c0 con 0x7fe6b0070e40 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): dump: 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3013401544 0 0) 0x7fe6b0073a60 con 0x7fe6b009fc40 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe69c004ec0 con 0x7fe6b009fc40 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe6a0001b80 con 0x7fe6b009fc40 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): dump: 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346285640 0 0) 0x7fe6b013abd0 con 0x7fe6b013c010 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient(hunting): my global_id is 23053 2024-09-16T03:14:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6b0073a60 con 0x7fe6b013c010 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2813638487 0 0) 0x7fe6b0073780 con 0x7fe6b0070e40 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 >> v1:172.21.15.45:6790/0 conn(0x7fe6b009fc40 legacy=0x7fe6b0070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 >> v1:172.21.15.46:6789/0 conn(0x7fe6b013c010 legacy=0x7fe6b0070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient: _finish_hunting 0 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 monclient: found mon.a 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b0071de0 con 0x7fe6b0070e40 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient: _finish_auth 0 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient: _check_auth_tickets 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 5 monclient: authenticate success, global_id 134130 2024-09-16T03:14:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: _renew_subs 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b0134ed0 con 0x7fe6b0070e40 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 client.134130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: _renew_subs 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6b01d6490 con 0x7fe6b0070e40 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 librados: init done 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6a8004b70 con 0x7fe6b0070e40 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe6a8004ec0 con 0x7fe6b0070e40 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6ae7fc640 4 set_mon_vals no callback set 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 monclient: dump: 2024-09-16T03:14:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe690005090 con 0x7fe6b0070e40 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fe6a8010f30 con 0x7fe6b0070e40 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(390..390 src has 99..390) ==== 5358+0+0 (unknown 4228477317 0 0) 0x7fe6a804ba20 con 0x7fe6b0070e40 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 10 client.134130.objecter ms_dispatch 0x7fe6b0000ba0 osd_map(390..390 src has 99..390) 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 3 client.134130.objecter handle_osd_map got epochs [390,390] > 0 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.159+0000 7fe6acff9640 3 client.134130.objecter handle_osd_map decoding full epoch 390 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.163+0000 7fe6acff9640 20 client.134130.objecter dump_active .. 0 homeless 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.163+0000 7fe6affff640 10 monclient: get_auth_request con 0x7fe69c03e210 auth_method 0 2024-09-16T03:14:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.163+0000 7fe6acff9640 10 client.134130.objecter ms_handle_connect 0x7fe69c03e210 2024-09-16T03:14:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.167+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fe6a801b080 con 0x7fe6b0070e40 2024-09-16T03:14:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.167+0000 7fe6acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.167+0000 7fe6acff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:29.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.399+0000 7fe6b65df640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:14:29.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.399+0000 7fe6b65df640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:29.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.399+0000 7fe6b65df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.399+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe690005380 con 0x7fe6b0070e40 2024-09-16T03:14:29.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6acff9640 1 -- 172.21.15.45:0/3938108938 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 262093682) 0x7fe6a801ffc0 con 0x7fe6b0070e40 2024-09-16T03:14:29.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:29.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6acff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:29.403 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:14:29.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 10 librados: watch_flush enter 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 10 librados: watch_flush exit 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 20 client.134130.objecter shutdown clearing up homeless session... 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 10 client.134130.objecter successfully canceled tick 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 >> v1:172.21.15.46:6812/149360280 conn(0x7fe69c03e210 legacy=0x7fe69c0406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 10 monclient: shutdown 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 >> v1:172.21.15.45:6789/0 conn(0x7fe6b0070e40 legacy=0x7fe6b01d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 shutdown_connections 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 >> 172.21.15.45:0/3938108938 conn(0x7fe6b00a2bd0 msgr2=0x7fe6b00a2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 shutdown_connections 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 1 -- 172.21.15.45:0/3938108938 wait complete. 2024-09-16T03:14:29.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.403+0000 7fe6b65df640 1 librados: shutdown 2024-09-16T03:14:29.422 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:14:29.423 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:14:29.423 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:14:29.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient: get_monmap_and_config 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient: build_initial_monmap 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient: monmap: 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:29.545023+0000 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:29.545023+0000 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:29.545 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 1 Processor -- start 2024-09-16T03:14:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 1 -- start start 2024-09-16T03:14:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient: init 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient(hunting): picked mon.noname-a con 0x7f0f200a25d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient(hunting): picked mon.noname-b con 0x7f0f20137340 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient(hunting): picked mon.noname-c con 0x7f0f2013ae50 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2009c3f0 con 0x7f0f200a25d0 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2009c6d0 con 0x7f0f2013ae50 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2009c9b0 con 0x7f0f20137340 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f27471640 10 monclient(hunting): authenticate will time out at 1560.195119s 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f24a0c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0f20137340 0x7f0f20139730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43814/0 (socket says 172.21.15.45:43814) 2024-09-16T03:14:29.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f24a0c640 1 -- 172.21.15.45:0/190629446 learned_addr learned my addr 172.21.15.45:0/190629446 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887970856 0 0) 0x7f0f2009c9b0 con 0x7f0f20137340 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient(hunting): my global_id is 124244 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f0c003530 con 0x7f0f20137340 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2395895759 0 0) 0x7f0f2009c3f0 con 0x7f0f200a25d0 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient(hunting): my global_id is 134133 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f2009c9b0 con 0x7f0f200a25d0 2024-09-16T03:14:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 552007169 0 0) 0x7f0f0c003530 con 0x7f0f20137340 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 >> v1:172.21.15.46:6789/0 conn(0x7f0f2013ae50 legacy=0x7f0f2013d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 >> v1:172.21.15.45:6789/0 conn(0x7f0f200a25d0 legacy=0x7f0f200a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient: _finish_hunting 0 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 monclient: found mon.noname-b 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2009b350 con 0x7f0f20137340 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient: _finish_auth 0 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient: _check_auth_tickets 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f100015d0 con 0x7f0f20137340 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.543+0000 7f0f17fff640 10 monclient: dump: 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 5 monclient: authenticate success, global_id 124244 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0f10001770 con 0x7f0f20137340 2024-09-16T03:14:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f17fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f17fff640 1 -- 172.21.15.45:0/190629446 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f10001b80 con 0x7f0f20137340 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f17fff640 10 monclient: dump: 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 4 set_mon_vals no callback set 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient: shutdown 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- 172.21.15.45:0/190629446 >> v1:172.21.15.45:6790/0 conn(0x7f0f20137340 legacy=0x7f0f20139730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- 172.21.15.45:0/190629446 shutdown_connections 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- 172.21.15.45:0/190629446 >> 172.21.15.45:0/190629446 conn(0x7f0f200a2bd0 msgr2=0x7f0f200a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- 172.21.15.45:0/190629446 shutdown_connections 2024-09-16T03:14:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- 172.21.15.45:0/190629446 wait complete. 2024-09-16T03:14:29.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient: build_initial_monmap 2024-09-16T03:14:29.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient: monmap: 2024-09-16T03:14:29.558 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:29.558 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:29.558 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:29.558 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:29.558 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:29.558 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 librados: starting msgr at 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 librados: starting objecter 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 Processor -- start 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- start start 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 librados: setting wanted keys 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 librados: calling monclient init 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient: init 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:14:29.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient(hunting): picked mon.noname-b con 0x7f0f200a25d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient(hunting): picked mon.noname-a con 0x7f0f20137340 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient(hunting): picked mon.noname-c con 0x7f0f2013ae50 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f20139b00 con 0x7f0f20137340 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2008e190 con 0x7f0f200a25d0 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f20138540 con 0x7f0f2013ae50 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient(hunting): authenticate will time out at 1560.198853s 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f24a0c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0f20137340 0x7f0f20137e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45318/0 (socket says 172.21.15.45:45318) 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f24a0c640 1 -- 172.21.15.45:0/3151701722 learned_addr learned my addr 172.21.15.45:0/3151701722 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f25a0e640 1 --1- 172.21.15.45:0/3151701722 >> v1:172.21.15.45:6790/0 conn(0x7f0f2013ae50 0x7f0f201d1a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43816/0 (socket says 172.21.15.45:43816) 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 client.?.objecter ms_handle_connect 0x7f0f20137340 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 client.?.objecter ms_handle_connect 0x7f0f2013ae50 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3651317836 0 0) 0x7f0f2009b350 con 0x7f0f20137340 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): my global_id is 134136 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f040070d0 con 0x7f0f20137340 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 client.?.objecter ms_handle_connect 0x7f0f200a25d0 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4183701084 0 0) 0x7f0f20138540 con 0x7f0f2013ae50 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): my global_id is 124247 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f2009b350 con 0x7f0f2013ae50 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3011579885 0 0) 0x7f0f040070d0 con 0x7f0f20137340 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f20138540 con 0x7f0f20137340 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f10001490 con 0x7f0f20137340 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): dump: 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 996495496 0 0) 0x7f0f2009b350 con 0x7f0f2013ae50 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f040070d0 con 0x7f0f2013ae50 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3473558869 0 0) 0x7f0f2008e190 con 0x7f0f200a25d0 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): my global_id is 23059 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f2009b350 con 0x7f0f200a25d0 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f18001840 con 0x7f0f2013ae50 2024-09-16T03:14:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient(hunting): dump: 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3405877602 0 0) 0x7f0f20138540 con 0x7f0f20137340 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 >> v1:172.21.15.45:6790/0 conn(0x7f0f2013ae50 legacy=0x7f0f201d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 >> v1:172.21.15.46:6789/0 conn(0x7f0f200a25d0 legacy=0x7f0f20137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 monclient: found mon.a 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2013e5e0 con 0x7f0f20137340 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient: _finish_auth 0 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 5 monclient: authenticate success, global_id 134136 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient: _renew_subs 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.547+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2013a560 con 0x7f0f20137340 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f27471640 10 client.134136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f27471640 10 monclient: _renew_subs 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f27471640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0f2013db60 con 0x7f0f20137340 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f27471640 1 librados: init done 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0f100016b0 con 0x7f0f20137340 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f10004c40 con 0x7f0f20137340 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient: dump: 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f177fe640 4 set_mon_vals no callback set 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f27471640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0f000065d0 con 0x7f0f00002870 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 client.134136.objecter ms_handle_connect 0x7f0f00002870 2024-09-16T03:14:29.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 client.134136.objecter resend_mon_ops 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f0f10010cb0 con 0x7f0f20137340 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(390..390 src has 99..390) ==== 5358+0+0 (unknown 4228477317 0 0) 0x7f0f1004b730 con 0x7f0f20137340 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 client.134136.objecter ms_dispatch 0x7f0f20000ba0 osd_map(390..390 src has 99..390) 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 3 client.134136.objecter handle_osd_map got epochs [390,390] > 0 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 3 client.134136.objecter handle_osd_map decoding full epoch 390 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 20 client.134136.objecter dump_active .. 0 homeless 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4077971307 0 0) 0x7f0f000065d0 con 0x7f0f00002870 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient(hunting): my global_id is 134139 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f20138540 con 0x7f0f00002870 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2834649200 0 0) 0x7f0f20138540 con 0x7f0f00002870 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f000065d0 con 0x7f0f00002870 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f25a0e640 10 monclient: get_auth_request con 0x7f0f04040420 auth_method 0 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f1c0015a0 con 0x7f0f00002870 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient: dump: 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.569 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1924204174 0 0) 0x7f0f000065d0 con 0x7f0f00002870 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f0000a170 con 0x7f0f00002870 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 client.134136.objecter ms_handle_connect 0x7f0f04040420 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0f1c002000 con 0x7f0f00002870 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 >> v1:172.21.15.45:6789/0 conn(0x7f0f00002870 legacy=0x7f0f00004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.551+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 >> v1:172.21.15.45:6789/0 conn(0x7f0f00002870 legacy=0x7f0f00004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0f00009e80 con 0x7f0f00009140 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 client.134136.objecter ms_handle_connect 0x7f0f00009140 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 client.134136.objecter resend_mon_ops 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366708347 0 0) 0x7f0f00009e80 con 0x7f0f00009140 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 monclient(hunting): my global_id is 134142 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f000065d0 con 0x7f0f00009140 2024-09-16T03:14:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1739625020 0 0) 0x7f0f000065d0 con 0x7f0f00009140 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f00009e80 con 0x7f0f00009140 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0f1c003b20 con 0x7f0f00009140 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 monclient: dump: 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.573 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 163664567 0 0) 0x7f0f00009e80 con 0x7f0f00009140 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0f0000c420 con 0x7f0f00009140 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3968 (unknown 0 0 3618883709) 0x7f0f1c004f50 con 0x7f0f00009140 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 >> v1:172.21.15.45:6789/0 conn(0x7f0f00009140 legacy=0x7f0f000095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f15ffb640 1 -- 172.21.15.45:0/3151701722 >> v1:172.21.15.45:6789/0 conn(0x7f0f00009140 legacy=0x7f0f000095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 40, 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 2787, 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:14:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:14:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:14:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:14:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:14:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:14:29.590 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:14:29.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 10 librados: watch_flush enter 2024-09-16T03:14:29.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 10 librados: watch_flush exit 2024-09-16T03:14:29.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 20 client.134136.objecter shutdown clearing up homeless session... 2024-09-16T03:14:29.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 10 client.134136.objecter successfully canceled tick 2024-09-16T03:14:29.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 >> v1:172.21.15.46:6812/149360280 conn(0x7f0f04040420 legacy=0x7f0f040428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 10 monclient: shutdown 2024-09-16T03:14:29.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:29.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 >> v1:172.21.15.45:6789/0 conn(0x7f0f20137340 legacy=0x7f0f20137e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f25a0e640 1 -- 172.21.15.45:0/3151701722 reap_dead start 2024-09-16T03:14:29.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f25a0e640 1 -- 172.21.15.45:0/3151701722 reap_dead start 2024-09-16T03:14:29.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 shutdown_connections 2024-09-16T03:14:29.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 >> 172.21.15.45:0/3151701722 conn(0x7f0f200a2bd0 msgr2=0x7f0f200a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:29.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 shutdown_connections 2024-09-16T03:14:29.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 1 -- 172.21.15.45:0/3151701722 wait complete. 2024-09-16T03:14:29.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.559+0000 7f0f27471640 1 librados: shutdown 2024-09-16T03:14:29.594 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:14:29.594 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:14:29.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:14:29.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient: get_monmap_and_config 2024-09-16T03:14:29.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient: build_initial_monmap 2024-09-16T03:14:29.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient: monmap: 2024-09-16T03:14:29.711 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:29.711 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.712 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:29.712446+0000 2024-09-16T03:14:29.712 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:29.712446+0000 2024-09-16T03:14:29.712 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:29.712 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.712 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:29.712 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:29.712 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:29.712 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 1 Processor -- start 2024-09-16T03:14:29.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 1 -- start start 2024-09-16T03:14:29.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient: init 2024-09-16T03:14:29.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:29.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:14:29.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient(hunting): picked mon.noname-b con 0x7f7c480a21c0 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:29.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient(hunting): picked mon.noname-c con 0x7f7c4809da10 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient(hunting): picked mon.noname-a con 0x7f7c4809d1a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4809ef60 con 0x7f7c4809d1a0 2024-09-16T03:14:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4809e6c0 con 0x7f7c4809da10 2024-09-16T03:14:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c480a4030 con 0x7f7c480a21c0 2024-09-16T03:14:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient(hunting): authenticate will time out at 1560.362462s 2024-09-16T03:14:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f7c480a21c0 0x7f7c480a25a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43828/0 (socket says 172.21.15.45:43828) 2024-09-16T03:14:29.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4659c640 1 -- 172.21.15.45:0/2452081675 learned_addr learned my addr 172.21.15.45:0/2452081675 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:29.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835213476 0 0) 0x7f7c480a4030 con 0x7f7c480a21c0 2024-09-16T03:14:29.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): my global_id is 124250 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c30003450 con 0x7f7c480a21c0 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 707057521 0 0) 0x7f7c4809ef60 con 0x7f7c4809d1a0 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): my global_id is 134145 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c480a4030 con 0x7f7c4809d1a0 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2806315113 0 0) 0x7f7c4809e6c0 con 0x7f7c4809da10 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient(hunting): my global_id is 23062 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c4809ef60 con 0x7f7c4809da10 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 186743965 0 0) 0x7f7c30003450 con 0x7f7c480a21c0 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 >> v1:172.21.15.46:6789/0 conn(0x7f7c4809da10 legacy=0x7f7c48139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 >> v1:172.21.15.45:6789/0 conn(0x7f7c4809d1a0 legacy=0x7f7c4813caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: _finish_hunting 0 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 monclient: found mon.noname-b 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c480a2cb0 con 0x7f7c480a21c0 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: _finish_auth 0 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: _check_auth_tickets 2024-09-16T03:14:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c380019a0 con 0x7f7c480a21c0 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: dump: 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.723 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 5 monclient: authenticate success, global_id 124250 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c38001b40 con 0x7f7c480a21c0 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 1 -- 172.21.15.45:0/2452081675 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c38001fd0 con 0x7f7c480a21c0 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4559a640 10 monclient: dump: 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.711+0000 7f7c4cdd2640 4 set_mon_vals no callback set 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: shutdown 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/2452081675 >> v1:172.21.15.45:6790/0 conn(0x7f7c480a21c0 legacy=0x7f7c480a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/2452081675 shutdown_connections 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/2452081675 >> 172.21.15.45:0/2452081675 conn(0x7f7c480a32b0 msgr2=0x7f7c480a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/2452081675 shutdown_connections 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/2452081675 wait complete. 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: build_initial_monmap 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: monmap: 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:29.726 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 librados: starting msgr at 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 librados: starting objecter 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 Processor -- start 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- start start 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 librados: setting wanted keys 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 librados: calling monclient init 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: init 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient(hunting): picked mon.noname-b con 0x7f7c4809d1a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient(hunting): picked mon.noname-c con 0x7f7c4809da10 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient(hunting): picked mon.noname-a con 0x7f7c480a21c0 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c481c67f0 con 0x7f7c480a21c0 2024-09-16T03:14:29.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c48124c80 con 0x7f7c4809d1a0 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c48074c90 con 0x7f7c4809da10 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient(hunting): authenticate will time out at 1560.365813s 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c45d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f7c4809da10 0x7f7c481cd6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43830/0 (socket says 172.21.15.45:43830) 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c45d9b640 1 -- 172.21.15.45:0/3287493663 learned_addr learned my addr 172.21.15.45:0/3287493663 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 client.?.objecter ms_handle_connect 0x7f7c4809da10 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 client.?.objecter ms_handle_connect 0x7f7c480a21c0 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2312225588 0 0) 0x7f7c40001900 con 0x7f7c480a21c0 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): my global_id is 134148 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c28004ec0 con 0x7f7c480a21c0 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326869162 0 0) 0x7f7c48074c90 con 0x7f7c4809da10 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): my global_id is 124253 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c40001900 con 0x7f7c4809da10 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 client.?.objecter ms_handle_connect 0x7f7c4809d1a0 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 813886707 0 0) 0x7f7c40001900 con 0x7f7c4809da10 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c48074c90 con 0x7f7c4809da10 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c3c001a00 con 0x7f7c4809da10 2024-09-16T03:14:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): dump: 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:29.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1800491189 0 0) 0x7f7c28004ec0 con 0x7f7c480a21c0 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c40001900 con 0x7f7c480a21c0 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1181969951 0 0) 0x7f7c48124c80 con 0x7f7c4809d1a0 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): my global_id is 23065 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c28004ec0 con 0x7f7c4809d1a0 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c400013c0 con 0x7f7c480a21c0 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient(hunting): dump: 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.731 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 365983172 0 0) 0x7f7c48074c90 con 0x7f7c4809da10 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 >> v1:172.21.15.46:6789/0 conn(0x7f7c4809d1a0 legacy=0x7f7c481344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 >> v1:172.21.15.45:6789/0 conn(0x7f7c480a21c0 legacy=0x7f7c481d0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 monclient: found mon.c 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c48123940 con 0x7f7c4809da10 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient: _finish_auth 0 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 5 monclient: authenticate success, global_id 124253 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: _renew_subs 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c481249a0 con 0x7f7c4809da10 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 client.124253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: _renew_subs 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c48134be0 con 0x7f7c4809da10 2024-09-16T03:14:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 librados: init done 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c3c001c60 con 0x7f7c4809da10 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c3c003a70 con 0x7f7c4809da10 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 monclient: dump: 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c44d99640 4 set_mon_vals no callback set 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7c2c0065d0 con 0x7f7c2c002870 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 client.124253.objecter ms_handle_connect 0x7f7c2c002870 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 10 client.124253.objecter resend_mon_ops 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.715+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7c3c003c90 con 0x7f7c4809da10 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(390..390 src has 99..390) ==== 5358+0+0 (unknown 4228477317 0 0) 0x7f7c3c04b730 con 0x7f7c4809da10 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 client.124253.objecter ms_dispatch 0x7f7c48000ba0 osd_map(390..390 src has 99..390) 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 3 client.124253.objecter handle_osd_map got epochs [390,390] > 0 2024-09-16T03:14:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 3 client.124253.objecter handle_osd_map decoding full epoch 390 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 20 client.124253.objecter dump_active .. 0 homeless 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4084440389 0 0) 0x7f7c2c0065d0 con 0x7f7c2c002870 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 monclient(hunting): my global_id is 124256 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c48074c90 con 0x7f7c2c002870 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2357782321 0 0) 0x7f7c48074c90 con 0x7f7c2c002870 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c2c0065d0 con 0x7f7c2c002870 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c38001860 con 0x7f7c2c002870 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 monclient: dump: 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c46d9d640 10 monclient: get_auth_request con 0x7f7c2803e180 auth_method 0 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1979079565 0 0) 0x7f7c2c0065d0 con 0x7f7c2c002870 2024-09-16T03:14:29.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c2c00a170 con 0x7f7c2c002870 2024-09-16T03:14:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 client.124253.objecter ms_handle_connect 0x7f7c2803e180 2024-09-16T03:14:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7c38001b40 con 0x7f7c2c002870 2024-09-16T03:14:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 >> v1:172.21.15.45:6790/0 conn(0x7f7c2c002870 legacy=0x7f7c2c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.719+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 >> v1:172.21.15.45:6790/0 conn(0x7f7c2c002870 legacy=0x7f7c2c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c4cdd2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:14:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7c2c009e80 con 0x7f7c2c009140 2024-09-16T03:14:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 client.124253.objecter ms_handle_connect 0x7f7c2c009140 2024-09-16T03:14:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 client.124253.objecter resend_mon_ops 2024-09-16T03:14:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 136414978 0 0) 0x7f7c2c009e80 con 0x7f7c2c009140 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 monclient(hunting): my global_id is 124259 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c2c0065d0 con 0x7f7c2c009140 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 562886259 0 0) 0x7f7c2c0065d0 con 0x7f7c2c009140 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c2c009e80 con 0x7f7c2c009140 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7c38005000 con 0x7f7c2c009140 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 monclient: dump: 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2085158796 0 0) 0x7f7c2c009e80 con 0x7f7c2c009140 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7c2c00c420 con 0x7f7c2c009140 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 871832385) 0x7f7c38006030 con 0x7f7c2c009140 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 >> v1:172.21.15.45:6790/0 conn(0x7f7c2c009140 legacy=0x7f7c2c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.723+0000 7f7c36ffd640 1 -- 172.21.15.45:0/3287493663 >> v1:172.21.15.45:6790/0 conn(0x7f7c2c009140 legacy=0x7f7c2c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:29.742 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:14:29.742 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:29.742 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:29.742 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:14:29.742 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 40, 2024-09-16T03:14:29.742 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 111954, 2024-09-16T03:14:29.742 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:14:29.742 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:29.743 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:29.744 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:14:29.745 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:29.746 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:29.747 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:14:29.748 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:29.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:29.751 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:29.752 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:14:29.753 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:14:29.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 10 librados: watch_flush enter 2024-09-16T03:14:29.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 10 librados: watch_flush exit 2024-09-16T03:14:29.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 20 client.124253.objecter shutdown clearing up homeless session... 2024-09-16T03:14:29.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 10 client.124253.objecter successfully canceled tick 2024-09-16T03:14:29.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 >> v1:172.21.15.46:6812/149360280 conn(0x7f7c2803e180 legacy=0x7f7c28040640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 10 monclient: shutdown 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 >> v1:172.21.15.45:6790/0 conn(0x7f7c4809da10 legacy=0x7f7c481cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c46d9d640 1 -- 172.21.15.45:0/3287493663 reap_dead start 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c46d9d640 1 -- 172.21.15.45:0/3287493663 reap_dead start 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 shutdown_connections 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 >> 172.21.15.45:0/3287493663 conn(0x7f7c480a32b0 msgr2=0x7f7c48070ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 shutdown_connections 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 1 -- 172.21.15.45:0/3287493663 wait complete. 2024-09-16T03:14:29.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.727+0000 7f7c4cdd2640 1 librados: shutdown 2024-09-16T03:14:29.756 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:14:29.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:14:29.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:14:29.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.871+0000 7ffa12f21640 10 monclient: get_monmap_and_config 2024-09-16T03:14:29.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.871+0000 7ffa12f21640 10 monclient: build_initial_monmap 2024-09-16T03:14:29.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.871+0000 7ffa12f21640 10 monclient: monmap: 2024-09-16T03:14:29.874 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:29.874 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.875 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:29.875344+0000 2024-09-16T03:14:29.875 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:29.875344+0000 2024-09-16T03:14:29.875 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:29.875 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.875 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:29.875 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:29.875 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:29.875 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 Processor -- start 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 -- start start 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient: init 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient(hunting): picked mon.noname-b con 0x7ffa0c0a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient(hunting): picked mon.noname-c con 0x7ffa0c0a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient(hunting): picked mon.noname-a con 0x7ffa0c073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0c09d9e0 con 0x7ffa0c073d40 2024-09-16T03:14:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0c09dc20 con 0x7ffa0c0a5810 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0c073360 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient(hunting): authenticate will time out at 1560.525332s 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa114be640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ffa0c073d40 0x7ffa0c140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45340/0 (socket says 172.21.15.45:45340) 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa10cbd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ffa0c0a4d20 0x7ffa0c0a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43852/0 (socket says 172.21.15.45:43852) 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa114be640 1 -- 172.21.15.45:0/2447385954 learned_addr learned my addr 172.21.15.45:0/2447385954 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1978407498 0 0) 0x7ffa0c073360 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient(hunting): my global_id is 124262 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9f4003530 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184161540 0 0) 0x7ffa0c09d9e0 con 0x7ffa0c073d40 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient(hunting): my global_id is 134151 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa0c073360 con 0x7ffa0c073d40 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2656942344 0 0) 0x7ff9f4003530 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 >> v1:172.21.15.46:6789/0 conn(0x7ffa0c0a5810 legacy=0x7ffa0c13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 >> v1:172.21.15.45:6789/0 conn(0x7ffa0c073d40 legacy=0x7ffa0c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 monclient: found mon.noname-b 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0c09cf30 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff9fc0016a0 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: dump: 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.881 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 5 monclient: authenticate success, global_id 124262 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9fc001840 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 1 -- 172.21.15.45:0/2447385954 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff9fc001cd0 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa0b7fe640 10 monclient: dump: 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.882 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 4 set_mon_vals no callback set 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient: shutdown 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 -- 172.21.15.45:0/2447385954 >> v1:172.21.15.45:6790/0 conn(0x7ffa0c0a4d20 legacy=0x7ffa0c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 -- 172.21.15.45:0/2447385954 shutdown_connections 2024-09-16T03:14:29.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 -- 172.21.15.45:0/2447385954 >> 172.21.15.45:0/2447385954 conn(0x7ffa0c0aa720 msgr2=0x7ffa0c0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:29.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 -- 172.21.15.45:0/2447385954 shutdown_connections 2024-09-16T03:14:29.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 1 -- 172.21.15.45:0/2447385954 wait complete. 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient: build_initial_monmap 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.875+0000 7ffa12f21640 10 monclient: monmap: 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 librados: starting msgr at 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 librados: starting objecter 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 Processor -- start 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 -- start start 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 librados: setting wanted keys 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 librados: calling monclient init 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient: init 2024-09-16T03:14:29.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient(hunting): picked mon.noname-a con 0x7ffa0c073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient(hunting): picked mon.noname-b con 0x7ffa0c0a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient(hunting): picked mon.noname-c con 0x7ffa0c0a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0c0a1510 con 0x7ffa0c073d40 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0c13f640 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0c13e150 con 0x7ffa0c0a5810 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa10cbd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ffa0c073d40 0x7ffa0c13c690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45344/0 (socket says 172.21.15.45:45344) 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa10cbd640 1 -- 172.21.15.45:0/2187279776 learned_addr learned my addr 172.21.15.45:0/2187279776 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient(hunting): authenticate will time out at 1560.528955s 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.?.objecter ms_handle_connect 0x7ffa0c073d40 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.?.objecter ms_handle_connect 0x7ffa0c0a5810 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1863428735 0 0) 0x7ffa0c0a1510 con 0x7ffa0c073d40 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): my global_id is 134154 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9f8004ec0 con 0x7ffa0c073d40 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596544846 0 0) 0x7ffa0c13e150 con 0x7ffa0c0a5810 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): my global_id is 124265 2024-09-16T03:14:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa0c0a1510 con 0x7ffa0c0a5810 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.?.objecter ms_handle_connect 0x7ffa0c0a4d20 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3799629403 0 0) 0x7ff9f8004ec0 con 0x7ffa0c073d40 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa0c13e150 con 0x7ffa0c073d40 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff9fc001cd0 con 0x7ffa0c073d40 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): dump: 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 237254619 0 0) 0x7ffa0c0a1510 con 0x7ffa0c0a5810 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9f8004ec0 con 0x7ffa0c0a5810 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ffa04001590 con 0x7ffa0c0a5810 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): dump: 2024-09-16T03:14:29.894 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3527862531 0 0) 0x7ffa0c13f640 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): my global_id is 23071 2024-09-16T03:14:29.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa0c0a1510 con 0x7ffa0c0a4d20 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1242873225 0 0) 0x7ffa0c13e150 con 0x7ffa0c073d40 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 >> v1:172.21.15.45:6790/0 conn(0x7ffa0c0a5810 legacy=0x7ffa0c1d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 >> v1:172.21.15.46:6789/0 conn(0x7ffa0c0a4d20 legacy=0x7ffa0c13cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: _finish_hunting 0 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 monclient: found mon.a 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0c1404b0 con 0x7ffa0c073d40 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: _finish_auth 0 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: _check_auth_tickets 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:29.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 5 monclient: authenticate success, global_id 134154 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient: _renew_subs 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0c13fc10 con 0x7ffa0c073d40 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 client.134154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient: _renew_subs 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ffa0c13ef10 con 0x7ffa0c073d40 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 librados: init done 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9fc001200 con 0x7ffa0c073d40 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff9fc004b40 con 0x7ffa0c073d40 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: dump: 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa0affd640 4 set_mon_vals no callback set 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:29.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff9ec0065d0 con 0x7ff9ec002870 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7ff9fc004d60 con 0x7ffa0c073d40 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(390..390 src has 99..390) ==== 5358+0+0 (unknown 4228477317 0 0) 0x7ff9fc04b860 con 0x7ffa0c073d40 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.134154.objecter ms_dispatch 0x7ffa0c000ba0 osd_map(390..390 src has 99..390) 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 3 client.134154.objecter handle_osd_map got epochs [390,390] > 0 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 3 client.134154.objecter handle_osd_map decoding full epoch 390 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 20 client.134154.objecter dump_active .. 0 homeless 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.134154.objecter ms_handle_connect 0x7ff9ec002870 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.134154.objecter resend_mon_ops 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa114be640 10 monclient: get_auth_request con 0x7ff9f803e2a0 auth_method 0 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 484925531 0 0) 0x7ff9ec0065d0 con 0x7ff9ec002870 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient(hunting): my global_id is 23074 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa0c13e150 con 0x7ff9ec002870 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 client.134154.objecter ms_handle_connect 0x7ff9f803e2a0 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3664912527 0 0) 0x7ffa0c13e150 con 0x7ff9ec002870 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9ec0065d0 con 0x7ff9ec002870 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ffa00001800 con 0x7ff9ec002870 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.879+0000 7ffa097fa640 10 monclient: dump: 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.900 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.883+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2834440340 0 0) 0x7ff9ec0065d0 con 0x7ff9ec002870 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.883+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9ec00a170 con 0x7ff9ec002870 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.883+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ffa000018d0 con 0x7ff9ec002870 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.883+0000 7ffa097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.883+0000 7ffa097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.883+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 >> v1:172.21.15.46:6789/0 conn(0x7ff9ec002870 legacy=0x7ff9ec004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.883+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 >> v1:172.21.15.46:6789/0 conn(0x7ff9ec002870 legacy=0x7ff9ec004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.887+0000 7ffa12f21640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.887+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff9ec009e80 con 0x7ff9ec009140 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.887+0000 7ffa097fa640 10 client.134154.objecter ms_handle_connect 0x7ff9ec009140 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.887+0000 7ffa097fa640 10 client.134154.objecter resend_mon_ops 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979882609 0 0) 0x7ff9ec009e80 con 0x7ff9ec009140 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 10 monclient(hunting): _negotiate 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 10 monclient(hunting): my global_id is 23077 2024-09-16T03:14:29.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9ec0065d0 con 0x7ff9ec009140 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 466268398 0 0) 0x7ff9ec0065d0 con 0x7ff9ec009140 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9ec009e80 con 0x7ff9ec009140 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ffa000037a0 con 0x7ff9ec009140 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 10 monclient: dump: 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:29.904 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1512280569 0 0) 0x7ff9ec009e80 con 0x7ff9ec009140 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff9ec00c400 con 0x7ff9ec009140 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 507775781) 0x7ffa000059c0 con 0x7ff9ec009140 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 >> v1:172.21.15.46:6789/0 conn(0x7ff9ec009140 legacy=0x7ff9ec0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa097fa640 1 -- 172.21.15.45:0/2187279776 >> v1:172.21.15.46:6789/0 conn(0x7ff9ec009140 legacy=0x7ff9ec0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:29.905 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:14:29.906 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:29.906 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:29.906 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:14:29.906 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 40, 2024-09-16T03:14:29.906 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 591693, 2024-09-16T03:14:29.906 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:14:29.906 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:14:29.906 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:14:29.906 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:29.907 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:29.908 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:14:29.909 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:29.910 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:29.911 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:29.912 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:29.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:29.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.915 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:29.916 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:29.917 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:29.918 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:29.918 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:29.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:29.918 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:14:29.918 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:14:29.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 10 librados: watch_flush enter 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 10 librados: watch_flush exit 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 20 client.134154.objecter shutdown clearing up homeless session... 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 10 client.134154.objecter successfully canceled tick 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 >> v1:172.21.15.46:6812/149360280 conn(0x7ff9f803e2a0 legacy=0x7ff9f8040760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 10 monclient: shutdown 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 >> v1:172.21.15.45:6789/0 conn(0x7ffa0c073d40 legacy=0x7ffa0c13c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa114be640 1 -- 172.21.15.45:0/2187279776 reap_dead start 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa114be640 1 -- 172.21.15.45:0/2187279776 reap_dead start 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 shutdown_connections 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 >> 172.21.15.45:0/2187279776 conn(0x7ffa0c0aa720 msgr2=0x7ffa0c0a40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 shutdown_connections 2024-09-16T03:14:29.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 1 -- 172.21.15.45:0/2187279776 wait complete. 2024-09-16T03:14:29.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:29.891+0000 7ffa12f21640 1 librados: shutdown 2024-09-16T03:14:29.921 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:14:29.921 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:14:29.921 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:14:29.921 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:14:29.921 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:14:29.921 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:14:29.922 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:14:29.934 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:14:29.934 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:14:29.934 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:14:30.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.087+0000 7f6060cc3640 10 monclient: get_monmap_and_config 2024-09-16T03:14:30.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.087+0000 7f6060cc3640 10 monclient: build_initial_monmap 2024-09-16T03:14:30.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.087+0000 7f6060cc3640 10 monclient: monmap: 2024-09-16T03:14:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:30.091159+0000 2024-09-16T03:14:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:30.091159+0000 2024-09-16T03:14:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:30.092 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:30.092 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:30.092 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:30.092 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.087+0000 7f6060cc3640 1 Processor -- start 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.087+0000 7f6060cc3640 1 -- start start 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.087+0000 7f6060cc3640 10 monclient: init 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): picked mon.noname-b con 0x7f605c135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): picked mon.noname-c con 0x7f605c09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): picked mon.noname-a con 0x7f605c13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c09f7e0 con 0x7f605c13c010 2024-09-16T03:14:30.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605a59c640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f605c135950 legacy=0x7f605c135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c09b160 con 0x7f605c09fc40 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c09b440 con 0x7f605c135950 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): authenticate will time out at 1560.740749s 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605ad9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f605c13c010 0x7f605c13e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45354/0 (socket says 172.21.15.45:45354) 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605ad9d640 1 -- 172.21.15.45:0/1419106171 learned_addr learned my addr 172.21.15.45:0/1419106171 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 -- 172.21.15.45:0/1419106171 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2825014428 0 0) 0x7f605c09f7e0 con 0x7f605c13c010 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient(hunting): _negotiate 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient(hunting): my global_id is 134157 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 -- 172.21.15.45:0/1419106171 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6044003530 con 0x7f605c13c010 2024-09-16T03:14:30.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 -- 172.21.15.45:0/1419106171 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 192825777 0 0) 0x7f6044003530 con 0x7f605c13c010 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 -- 172.21.15.45:0/1419106171 >> v1:172.21.15.45:6790/0 conn(0x7f605c135950 legacy=0x7f605c135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 -- 172.21.15.45:0/1419106171 >> v1:172.21.15.46:6789/0 conn(0x7f605c09fc40 legacy=0x7f605c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: _finish_hunting 0 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 monclient: found mon.noname-a 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 -- 172.21.15.45:0/1419106171 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c136440 con 0x7f605c13c010 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: _finish_auth 0 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: _check_auth_tickets 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 -- 172.21.15.45:0/1419106171 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6054001a20 con 0x7f605c13c010 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:30.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: dump: 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 5 monclient: authenticate success, global_id 134157 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:30.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 -- 172.21.15.45:0/1419106171 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6054001c40 con 0x7f605c13c010 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 4 set_mon_vals no callback set 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 1 -- 172.21.15.45:0/1419106171 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60540020d0 con 0x7f605c13c010 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605959a640 10 monclient: dump: 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:30.108 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient: shutdown 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- 172.21.15.45:0/1419106171 >> v1:172.21.15.45:6789/0 conn(0x7f605c13c010 legacy=0x7f605c13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- 172.21.15.45:0/1419106171 shutdown_connections 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- 172.21.15.45:0/1419106171 >> 172.21.15.45:0/1419106171 conn(0x7f605c0a2bd0 msgr2=0x7f605c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- 172.21.15.45:0/1419106171 shutdown_connections 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- 172.21.15.45:0/1419106171 wait complete. 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient: build_initial_monmap 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient: monmap: 2024-09-16T03:14:30.109 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 librados: starting msgr at 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 librados: starting objecter 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 Processor -- start 2024-09-16T03:14:30.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- start start 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 librados: setting wanted keys 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 librados: calling monclient init 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient: init 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): picked mon.noname-c con 0x7f605c09fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): picked mon.noname-a con 0x7f605c135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): picked mon.noname-b con 0x7f605c13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c1348f0 con 0x7f605c135950 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c09d2c0 con 0x7f605c13c010 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c09ed50 con 0x7f605c09fc40 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:30.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f605a59c640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f605c09fc40 legacy=0x7f605c0727c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6060cc3640 10 monclient(hunting): authenticate will time out at 1560.743635s 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6059d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f605c135950 0x7f605c072ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45358/0 (socket says 172.21.15.45:45358) 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f6059d9b640 1 -- 172.21.15.45:0/1598899786 learned_addr learned my addr 172.21.15.45:0/1598899786 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f604affd640 10 client.?.objecter ms_handle_connect 0x7f605c135950 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f604affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159376561 0 0) 0x7f605c1348f0 con 0x7f605c135950 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f604affd640 10 monclient(hunting): _negotiate 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f604affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f604affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.091+0000 7f604affd640 10 monclient(hunting): my global_id is 134160 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f603c004ec0 con 0x7f605c135950 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 client.?.objecter ms_handle_connect 0x7f605c13c010 2024-09-16T03:14:30.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3151243179 0 0) 0x7f603c004ec0 con 0x7f605c135950 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f605c1348f0 con 0x7f605c135950 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6050001a00 con 0x7f605c135950 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient(hunting): dump: 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2891736604 0 0) 0x7f605c1348f0 con 0x7f605c135950 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 >> v1:172.21.15.45:6790/0 conn(0x7f605c09fc40 legacy=0x7f605c0727c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 >> v1:172.21.15.46:6789/0 conn(0x7f605c13c010 legacy=0x7f605c1d4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient: _finish_hunting 0 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 monclient: found mon.a 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c09f030 con 0x7f605c135950 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient: _finish_auth 0 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient: _check_auth_tickets 2024-09-16T03:14:30.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 5 monclient: authenticate success, global_id 134160 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 10 monclient: _renew_subs 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f605c0735e0 con 0x7f605c135950 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 10 client.134160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 10 monclient: _renew_subs 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f605c13b310 con 0x7f605c135950 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 1 librados: init done 2024-09-16T03:14:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6050001d40 con 0x7f605c135950 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6050003a70 con 0x7f605c135950 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 monclient: dump: 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6058d99640 4 set_mon_vals no callback set 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f6050003c10 con 0x7f605c135950 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6040005090 con 0x7f605c135950 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(390..390 src has 99..390) ==== 5358+0+0 (unknown 4228477317 0 0) 0x7f605004b910 con 0x7f605c135950 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 client.134160.objecter ms_dispatch 0x7f605c000ba0 osd_map(390..390 src has 99..390) 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 3 client.134160.objecter handle_osd_map got epochs [390,390] > 0 2024-09-16T03:14:30.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 3 client.134160.objecter handle_osd_map decoding full epoch 390 2024-09-16T03:14:30.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 20 client.134160.objecter dump_active .. 0 homeless 2024-09-16T03:14:30.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f605a59c640 10 monclient: get_auth_request con 0x7f603c03ded0 auth_method 0 2024-09-16T03:14:30.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.095+0000 7f604affd640 10 client.134160.objecter ms_handle_connect 0x7f603c03ded0 2024-09-16T03:14:30.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.099+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f6050016c10 con 0x7f605c135950 2024-09-16T03:14:30.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.099+0000 7f604affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:30.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.099+0000 7f604affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.335+0000 7f6060cc3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:14:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.335+0000 7f6060cc3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.335+0000 7f6060cc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.335+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6040005380 con 0x7f605c135950 2024-09-16T03:14:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f604affd640 1 -- 172.21.15.45:0/1598899786 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 4291061725) 0x7f6050016510 con 0x7f605c135950 2024-09-16T03:14:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f604affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f604affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:30.340 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:14:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 10 librados: watch_flush enter 2024-09-16T03:14:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 10 librados: watch_flush exit 2024-09-16T03:14:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 20 client.134160.objecter shutdown clearing up homeless session... 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 10 client.134160.objecter successfully canceled tick 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 >> v1:172.21.15.46:6812/149360280 conn(0x7f603c03ded0 legacy=0x7f603c040390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 10 monclient: shutdown 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 >> v1:172.21.15.45:6789/0 conn(0x7f605c135950 legacy=0x7f605c072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 shutdown_connections 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 >> 172.21.15.45:0/1598899786 conn(0x7f605c0a2bd0 msgr2=0x7f605c13eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 shutdown_connections 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 1 -- 172.21.15.45:0/1598899786 wait complete. 2024-09-16T03:14:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:30.339+0000 7f6060cc3640 1 librados: shutdown 2024-09-16T03:14:33.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient: get_monmap_and_config 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient: build_initial_monmap 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient: monmap: 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:33.489924+0000 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:33.489924+0000 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:33.490 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 1 Processor -- start 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 1 -- start start 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient: init 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient(hunting): picked mon.noname-a con 0x7f94a0135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient(hunting): picked mon.noname-b con 0x7f94a009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient(hunting): picked mon.noname-c con 0x7f94a013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a009f7e0 con 0x7f94a0135950 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a009b160 con 0x7f94a013c010 2024-09-16T03:14:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a009b440 con 0x7f94a009fc40 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a772a640 10 monclient(hunting): authenticate will time out at 1564.139231s 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a4cc5640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f94a009fc40 legacy=0x7f94a013a8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a54c6640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f94a0135950 0x7f94a0135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50182/0 (socket says 172.21.15.45:50182) 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f94a54c6640 1 -- 172.21.15.45:0/2756483316 learned_addr learned my addr 172.21.15.45:0/2756483316 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 1 -- 172.21.15.45:0/2756483316 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1386325279 0 0) 0x7f94a009f7e0 con 0x7f94a0135950 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 10 monclient(hunting): _negotiate 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 10 monclient(hunting): my global_id is 134163 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 1 -- 172.21.15.45:0/2756483316 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f948c003530 con 0x7f94a0135950 2024-09-16T03:14:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 1 -- 172.21.15.45:0/2756483316 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 390338936 0 0) 0x7f948c003530 con 0x7f94a0135950 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 1 -- 172.21.15.45:0/2756483316 >> v1:172.21.15.45:6790/0 conn(0x7f94a009fc40 legacy=0x7f94a013a8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 1 -- 172.21.15.45:0/2756483316 >> v1:172.21.15.46:6789/0 conn(0x7f94a013c010 legacy=0x7f94a013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 10 monclient: _finish_hunting 0 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 1 monclient: found mon.noname-a 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.487+0000 7f9497fff640 1 -- 172.21.15.45:0/2756483316 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a0136440 con 0x7f94a0135950 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 10 monclient: _finish_auth 0 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 10 monclient: _check_auth_tickets 2024-09-16T03:14:33.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 1 -- 172.21.15.45:0/2756483316 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f949c0016a0 con 0x7f94a0135950 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 10 monclient: dump: 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:33.499 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 5 monclient: authenticate success, global_id 134163 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 1 -- 172.21.15.45:0/2756483316 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f949c0018c0 con 0x7f94a0135950 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 1 -- 172.21.15.45:0/2756483316 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f949c001d50 con 0x7f94a0135950 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9497fff640 10 monclient: dump: 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 4 set_mon_vals no callback set 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: shutdown 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- 172.21.15.45:0/2756483316 >> v1:172.21.15.45:6789/0 conn(0x7f94a0135950 legacy=0x7f94a0135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- 172.21.15.45:0/2756483316 shutdown_connections 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- 172.21.15.45:0/2756483316 >> 172.21.15.45:0/2756483316 conn(0x7f94a00a2bd0 msgr2=0x7f94a00a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:33.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- 172.21.15.45:0/2756483316 shutdown_connections 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- 172.21.15.45:0/2756483316 wait complete. 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: build_initial_monmap 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: monmap: 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:33.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 librados: starting msgr at 2024-09-16T03:14:33.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 librados: starting objecter 2024-09-16T03:14:33.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 Processor -- start 2024-09-16T03:14:33.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- start start 2024-09-16T03:14:33.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 librados: setting wanted keys 2024-09-16T03:14:33.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 librados: calling monclient init 2024-09-16T03:14:33.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: init 2024-09-16T03:14:33.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:33.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:14:33.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient(hunting): picked mon.noname-a con 0x7f94a009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient(hunting): picked mon.noname-c con 0x7f94a0135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient(hunting): picked mon.noname-b con 0x7f94a013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a013bab0 con 0x7f94a009fc40 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a009ede0 con 0x7f94a013c010 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a009f0c0 con 0x7f94a0135950 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient(hunting): authenticate will time out at 1564.142190s 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a4cc5640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f94a0135950 legacy=0x7f94a01d1d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a54c6640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f94a009fc40 0x7f94a01ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50186/0 (socket says 172.21.15.45:50186) 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a54c6640 1 -- 172.21.15.45:0/3913093889 learned_addr learned my addr 172.21.15.45:0/3913093889 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 client.?.objecter ms_handle_connect 0x7f94a009fc40 2024-09-16T03:14:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 726044127 0 0) 0x7f94a0136440 con 0x7f94a009fc40 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient(hunting): my global_id is 134166 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9484004ec0 con 0x7f94a009fc40 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 client.?.objecter ms_handle_connect 0x7f94a013c010 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 689326198 0 0) 0x7f9484004ec0 con 0x7f94a009fc40 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94a0136440 con 0x7f94a009fc40 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f949c001c10 con 0x7f94a009fc40 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient(hunting): dump: 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1731492193 0 0) 0x7f94a0136440 con 0x7f94a009fc40 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 >> v1:172.21.15.45:6790/0 conn(0x7f94a0135950 legacy=0x7f94a01d1d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 >> v1:172.21.15.46:6789/0 conn(0x7f94a013c010 legacy=0x7f94a01d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 monclient: found mon.a 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a009d640 con 0x7f94a009fc40 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient: _finish_auth 0 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 5 monclient: authenticate success, global_id 134166 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: _renew_subs 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:33.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f94a009d8c0 con 0x7f94a009fc40 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 client.134166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: _renew_subs 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f94a00737d0 con 0x7f94a009fc40 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 librados: init done 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f949c001820 con 0x7f94a009fc40 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9496ffd640 4 set_mon_vals no callback set 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f949c004e60 con 0x7f94a009fc40 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:33.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f9495ffb640 10 monclient: dump: 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.491+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9480005090 con 0x7f94a009fc40 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.495+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f949c010ed0 con 0x7f94a009fc40 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.495+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(390..390 src has 99..390) ==== 5358+0+0 (unknown 4228477317 0 0) 0x7f949c04b930 con 0x7f94a009fc40 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.495+0000 7f9495ffb640 10 client.134166.objecter ms_dispatch 0x7f94a0000ba0 osd_map(390..390 src has 99..390) 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.495+0000 7f9495ffb640 3 client.134166.objecter handle_osd_map got epochs [390,390] > 0 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.495+0000 7f9495ffb640 3 client.134166.objecter handle_osd_map decoding full epoch 390 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.495+0000 7f9495ffb640 20 client.134166.objecter dump_active .. 0 homeless 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.495+0000 7f94a4cc5640 10 monclient: get_auth_request con 0x7f948403df20 auth_method 0 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.495+0000 7f9495ffb640 10 client.134166.objecter ms_handle_connect 0x7f948403df20 2024-09-16T03:14:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.499+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f94a013bab0 con 0x7f94a009fc40 2024-09-16T03:14:33.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.499+0000 7f9495ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:33.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.499+0000 7f9495ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:14:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9480005380 con 0x7f94a009fc40 2024-09-16T03:14:33.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f9495ffb640 1 -- 172.21.15.45:0/3913093889 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 3234255581) 0x7f949c016cf0 con 0x7f94a009fc40 2024-09-16T03:14:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f9495ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:33.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f9495ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:33.729 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 10 librados: watch_flush enter 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 10 librados: watch_flush exit 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 20 client.134166.objecter shutdown clearing up homeless session... 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 10 client.134166.objecter successfully canceled tick 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 >> v1:172.21.15.46:6812/149360280 conn(0x7f948403df20 legacy=0x7f94840403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 10 monclient: shutdown 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 >> v1:172.21.15.45:6789/0 conn(0x7f94a009fc40 legacy=0x7f94a01ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 shutdown_connections 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 >> 172.21.15.45:0/3913093889 conn(0x7f94a00a2bd0 msgr2=0x7f94a013ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 shutdown_connections 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 1 -- 172.21.15.45:0/3913093889 wait complete. 2024-09-16T03:14:33.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:33.727+0000 7f94a772a640 1 librados: shutdown 2024-09-16T03:14:36.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:14:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient: get_monmap_and_config 2024-09-16T03:14:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient: build_initial_monmap 2024-09-16T03:14:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient: monmap: 2024-09-16T03:14:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:36.878110+0000 2024-09-16T03:14:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:36.878110+0000 2024-09-16T03:14:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:36.878 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:36.878 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:36.878 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:36.878 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 1 Processor -- start 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 1 -- start start 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient: init 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient(hunting): picked mon.noname-c con 0x7fd71c135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient(hunting): picked mon.noname-a con 0x7fd71c09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient(hunting): picked mon.noname-b con 0x7fd71c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c09f7e0 con 0x7fd71c09fc40 2024-09-16T03:14:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c09b160 con 0x7fd71c135950 2024-09-16T03:14:36.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c09b440 con 0x7fd71c13c010 2024-09-16T03:14:36.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:36.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd721edf640 10 monclient(hunting): authenticate will time out at 1567.527466s 2024-09-16T03:14:36.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd71bfff640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd71c13e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:36.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd71affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd71c09fc40 0x7fd71c13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50190/0 (socket says 172.21.15.45:50190) 2024-09-16T03:14:36.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:36.875+0000 7fd71affd640 1 -- 172.21.15.45:0/4233984132 learned_addr learned my addr 172.21.15.45:0/4233984132 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:37.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:37.079+0000 7fd71bfff640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd71c13e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:37.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:37.483+0000 7fd71bfff640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd71c13e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:38.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:38.287+0000 7fd71bfff640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd71c13e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 10 monclient(hunting): tick 2024-09-16T03:14:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:14:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 1 monclient(hunting): continuing hunt 2024-09-16T03:14:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:14:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd71c13e400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:14:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.46:6789/0 conn(0x7fd71c135950 legacy=0x7fd71c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6789/0 conn(0x7fd71c09fc40 legacy=0x7fd71c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:14:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 10 monclient(hunting): picked mon.noname-c con 0x7fd7080036d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 10 monclient(hunting): picked mon.noname-a con 0x7fd708007390 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 10 monclient(hunting): picked mon.noname-b con 0x7fd71c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c136440 con 0x7fd708007390 2024-09-16T03:14:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c09b440 con 0x7fd7080036d0 2024-09-16T03:14:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd708010930 con 0x7fd71c13c010 2024-09-16T03:14:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd7197fa640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:39.875+0000 7fd71bfff640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd70800d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:40.079+0000 7fd71bfff640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd70800d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:40.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:40.483+0000 7fd71bfff640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd70800d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:41.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:41.287+0000 7fd71bfff640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd70800d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:42.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.875+0000 7fd7197fa640 10 monclient(hunting): tick 2024-09-16T03:14:42.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.875+0000 7fd7197fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:14:42.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.875+0000 7fd7197fa640 1 monclient(hunting): continuing hunt 2024-09-16T03:14:42.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.875+0000 7fd7197fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:14:42.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.875+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd70800d110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.875+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.46:6789/0 conn(0x7fd7080036d0 legacy=0x7fd708005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.875+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6789/0 conn(0x7fd708007390 legacy=0x7fd708009800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.875+0000 7fd7197fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.879+0000 7fd7197fa640 10 monclient(hunting): picked mon.noname-c con 0x7fd708010c30 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.879+0000 7fd7197fa640 10 monclient(hunting): picked mon.noname-b con 0x7fd71c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.879+0000 7fd7197fa640 10 monclient(hunting): picked mon.noname-a con 0x7fd708012e90 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.879+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd708011bd0 con 0x7fd708012e90 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.879+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd708010930 con 0x7fd708010c30 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.879+0000 7fd7197fa640 1 -- 172.21.15.45:0/4233984132 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd708003b10 con 0x7fd71c13c010 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.879+0000 7fd7197fa640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:42.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:42.879+0000 7fd71affd640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd70800d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:43.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:43.083+0000 7fd71affd640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd70800d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:43.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:43.487+0000 7fd71affd640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd70800d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:44.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.179+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418106465 0 0) 0x7fd708011bd0 con 0x7fd708012e90 2024-09-16T03:14:44.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.179+0000 7fd71a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.179+0000 7fd71a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.179+0000 7fd71a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.179+0000 7fd71a7fc640 10 monclient(hunting): my global_id is 134169 2024-09-16T03:14:44.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.179+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd704003530 con 0x7fd708012e90 2024-09-16T03:14:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3473425693 0 0) 0x7fd708010930 con 0x7fd708010c30 2024-09-16T03:14:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient(hunting): my global_id is 23080 2024-09-16T03:14:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd708011bd0 con 0x7fd708010c30 2024-09-16T03:14:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1279104989 0 0) 0x7fd704003530 con 0x7fd708012e90 2024-09-16T03:14:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6790/0 conn(0x7fd71c13c010 legacy=0x7fd70800d110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:14:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.46:6789/0 conn(0x7fd708010c30 legacy=0x7fd708011010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:14:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 monclient: found mon.noname-a 2024-09-16T03:14:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd708004cf0 con 0x7fd708012e90 2024-09-16T03:14:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:14:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:14:44.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:44.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd714004600 con 0x7fd708012e90 2024-09-16T03:14:44.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:44.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: dump: 2024-09-16T03:14:44.186 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.186 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd714004810 con 0x7fd708012e90 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:44.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 1 -- 172.21.15.45:0/4233984132 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd714004c60 con 0x7fd708012e90 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd71a7fc640 10 monclient: dump: 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.188 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 5 monclient: authenticate success, global_id 134169 2024-09-16T03:14:44.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:44.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 4 set_mon_vals no callback set 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 10 monclient: shutdown 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 1 -- 172.21.15.45:0/4233984132 >> v1:172.21.15.45:6789/0 conn(0x7fd708012e90 legacy=0x7fd708015280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 1 -- 172.21.15.45:0/4233984132 shutdown_connections 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 1 -- 172.21.15.45:0/4233984132 >> 172.21.15.45:0/4233984132 conn(0x7fd71c0a2bd0 msgr2=0x7fd71c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 1 -- 172.21.15.45:0/4233984132 shutdown_connections 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.183+0000 7fd721edf640 1 -- 172.21.15.45:0/4233984132 wait complete. 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient: build_initial_monmap 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient: monmap: 2024-09-16T03:14:44.191 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 1 librados: starting msgr at 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 1 librados: starting objecter 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 1 Processor -- start 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 1 -- start start 2024-09-16T03:14:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 1 librados: setting wanted keys 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 1 librados: calling monclient init 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient: init 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient(hunting): picked mon.noname-c con 0x7fd708010c30 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient(hunting): picked mon.noname-a con 0x7fd708012e90 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient(hunting): picked mon.noname-b con 0x7fd71c13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c12bfb0 con 0x7fd708012e90 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c134e30 con 0x7fd71c13c010 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c09d2c0 con 0x7fd708010c30 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd721edf640 10 monclient(hunting): authenticate will time out at 1574.838090s 2024-09-16T03:14:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd71b7fe640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fd708010c30 legacy=0x7fd71c12b190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd71affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd708012e90 0x7fd71c12b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54524/0 (socket says 172.21.15.45:54524) 2024-09-16T03:14:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd71affd640 1 -- 172.21.15.45:0/1968940895 learned_addr learned my addr 172.21.15.45:0/1968940895 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd703fff640 10 client.?.objecter ms_handle_connect 0x7fd708012e90 2024-09-16T03:14:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd703fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd703fff640 10 client.?.objecter ms_handle_connect 0x7fd71c13c010 2024-09-16T03:14:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.187+0000 7fd703fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808799219 0 0) 0x7fd71c12bfb0 con 0x7fd708012e90 2024-09-16T03:14:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient(hunting): my global_id is 134172 2024-09-16T03:14:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7080117c0 con 0x7fd708012e90 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2675499081 0 0) 0x7fd71c136440 con 0x7fd708012e90 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd71c12bfb0 con 0x7fd708012e90 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd710001770 con 0x7fd708012e90 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient(hunting): dump: 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3826294095 0 0) 0x7fd71c12bfb0 con 0x7fd708012e90 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 >> v1:172.21.15.45:6790/0 conn(0x7fd708010c30 legacy=0x7fd71c12b190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 >> v1:172.21.15.46:6789/0 conn(0x7fd71c13c010 legacy=0x7fd71c1d5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient: _finish_hunting 0 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 monclient: found mon.a 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c1350d0 con 0x7fd708012e90 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient: _finish_auth 0 2024-09-16T03:14:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient: _check_auth_tickets 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd721edf640 5 monclient: authenticate success, global_id 134172 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd721edf640 10 monclient: _renew_subs 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd721edf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd71c12c290 con 0x7fd708012e90 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd721edf640 10 client.134172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd721edf640 10 monclient: _renew_subs 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd721edf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd71c12dd50 con 0x7fd708012e90 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd721edf640 1 librados: init done 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd710001d40 con 0x7fd708012e90 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd7100049d0 con 0x7fd708012e90 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd703fff640 10 monclient: dump: 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.200 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.191+0000 7fd719ffb640 4 set_mon_vals no callback set 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd721edf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd721edf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd721edf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6fc005090 con 0x7fd708012e90 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fd710004c10 con 0x7fd708012e90 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(391..391 src has 99..391) ==== 5374+0+0 (unknown 589942475 0 0) 0x7fd71004b740 con 0x7fd708012e90 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd703fff640 10 client.134172.objecter ms_dispatch 0x7fd71c000ba0 osd_map(391..391 src has 99..391) 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd703fff640 3 client.134172.objecter handle_osd_map got epochs [391,391] > 0 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd703fff640 3 client.134172.objecter handle_osd_map decoding full epoch 391 2024-09-16T03:14:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd703fff640 20 client.134172.objecter dump_active .. 0 homeless 2024-09-16T03:14:44.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd71b7fe640 10 monclient: get_auth_request con 0x7fd70803f850 auth_method 0 2024-09-16T03:14:44.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.195+0000 7fd703fff640 10 client.134172.objecter ms_handle_connect 0x7fd70803f850 2024-09-16T03:14:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.203+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd7080117c0 con 0x7fd708012e90 2024-09-16T03:14:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.203+0000 7fd703fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.203+0000 7fd703fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:44.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.451+0000 7fd721edf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:14:44.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.451+0000 7fd721edf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:44.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.451+0000 7fd721edf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.451+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd6fc005380 con 0x7fd708012e90 2024-09-16T03:14:44.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd703fff640 1 -- 172.21.15.45:0/1968940895 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3266655683) 0x7fd710016a30 con 0x7fd708012e90 2024-09-16T03:14:44.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd703fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:14:44.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd703fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:44.456 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:14:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 10 librados: watch_flush enter 2024-09-16T03:14:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 10 librados: watch_flush exit 2024-09-16T03:14:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 20 client.134172.objecter shutdown clearing up homeless session... 2024-09-16T03:14:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 10 client.134172.objecter successfully canceled tick 2024-09-16T03:14:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 >> v1:172.21.15.46:6812/149360280 conn(0x7fd70803f850 legacy=0x7fd708041d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 10 monclient: shutdown 2024-09-16T03:14:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 >> v1:172.21.15.45:6789/0 conn(0x7fd708012e90 legacy=0x7fd71c12b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 shutdown_connections 2024-09-16T03:14:44.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 >> 172.21.15.45:0/1968940895 conn(0x7fd71c0a2bd0 msgr2=0x7fd71c0a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:44.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 shutdown_connections 2024-09-16T03:14:44.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 1 -- 172.21.15.45:0/1968940895 wait complete. 2024-09-16T03:14:44.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.455+0000 7fd721edf640 1 librados: shutdown 2024-09-16T03:14:44.473 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:14:44.473 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:14:44.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:14:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient: get_monmap_and_config 2024-09-16T03:14:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient: build_initial_monmap 2024-09-16T03:14:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient: monmap: 2024-09-16T03:14:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:44.613177+0000 2024-09-16T03:14:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:44.613177+0000 2024-09-16T03:14:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:44.614 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 1 Processor -- start 2024-09-16T03:14:44.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 1 -- start start 2024-09-16T03:14:44.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient: init 2024-09-16T03:14:44.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:44.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:14:44.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient(hunting): picked mon.noname-b con 0x7fdbac135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:44.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient(hunting): picked mon.noname-a con 0x7fdbac09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:44.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient(hunting): picked mon.noname-c con 0x7fdbac13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:44.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbac09f7e0 con 0x7fdbac09fc40 2024-09-16T03:14:44.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbac09b160 con 0x7fdbac13c010 2024-09-16T03:14:44.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbac09b440 con 0x7fdbac135950 2024-09-16T03:14:44.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:44.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient(hunting): authenticate will time out at 1575.262544s 2024-09-16T03:14:44.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbab7fe640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fdbac135950 legacy=0x7fdbac135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:44.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaaffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdbac09fc40 0x7fdbac13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54528/0 (socket says 172.21.15.45:54528) 2024-09-16T03:14:44.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaaffd640 1 -- 172.21.15.45:0/2463358548 learned_addr learned my addr 172.21.15.45:0/2463358548 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:44.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 -- 172.21.15.45:0/2463358548 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 35022713 0 0) 0x7fdbac09f7e0 con 0x7fdbac09fc40 2024-09-16T03:14:44.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient(hunting): my global_id is 134175 2024-09-16T03:14:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 -- 172.21.15.45:0/2463358548 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb98003530 con 0x7fdbac09fc40 2024-09-16T03:14:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 -- 172.21.15.45:0/2463358548 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3034417493 0 0) 0x7fdb98003530 con 0x7fdbac09fc40 2024-09-16T03:14:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 -- 172.21.15.45:0/2463358548 >> v1:172.21.15.45:6790/0 conn(0x7fdbac135950 legacy=0x7fdbac135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:14:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 -- 172.21.15.45:0/2463358548 >> v1:172.21.15.46:6789/0 conn(0x7fdbac13c010 legacy=0x7fdbac13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:14:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 monclient: found mon.noname-a 2024-09-16T03:14:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 -- 172.21.15.45:0/2463358548 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac136440 con 0x7fdbac09fc40 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: _finish_auth 0 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 -- 172.21.15.45:0/2463358548 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdb940016a0 con 0x7fdbac09fc40 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: dump: 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.634 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 5 monclient: authenticate success, global_id 134175 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 -- 172.21.15.45:0/2463358548 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb940018c0 con 0x7fdbac09fc40 2024-09-16T03:14:44.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 1 -- 172.21.15.45:0/2463358548 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdb94001d50 con 0x7fdbac09fc40 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 4 set_mon_vals no callback set 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbaa7fc640 10 monclient: dump: 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.636 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 10 monclient: shutdown 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.611+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2463358548 >> v1:172.21.15.45:6789/0 conn(0x7fdbac09fc40 legacy=0x7fdbac13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2463358548 shutdown_connections 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2463358548 >> 172.21.15.45:0/2463358548 conn(0x7fdbac0a2bd0 msgr2=0x7fdbac0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:44.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2463358548 shutdown_connections 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2463358548 wait complete. 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient: build_initial_monmap 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient: monmap: 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.638 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:44.639 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:44.639 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:44.639 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 librados: starting msgr at 2024-09-16T03:14:44.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 librados: starting objecter 2024-09-16T03:14:44.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 Processor -- start 2024-09-16T03:14:44.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- start start 2024-09-16T03:14:44.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 librados: setting wanted keys 2024-09-16T03:14:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 librados: calling monclient init 2024-09-16T03:14:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient: init 2024-09-16T03:14:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:14:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient(hunting): picked mon.noname-a con 0x7fdbac09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient(hunting): picked mon.noname-c con 0x7fdbac135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient(hunting): picked mon.noname-b con 0x7fdbac13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbac13bab0 con 0x7fdbac09fc40 2024-09-16T03:14:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbac09e5d0 con 0x7fdbac13c010 2024-09-16T03:14:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbac09e8b0 con 0x7fdbac135950 2024-09-16T03:14:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient(hunting): authenticate will time out at 1575.265775s 2024-09-16T03:14:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbaaffd640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fdbac135950 legacy=0x7fdbac072ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbab7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdbac09fc40 0x7fdbac0727c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54540/0 (socket says 172.21.15.45:54540) 2024-09-16T03:14:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbab7fe640 1 -- 172.21.15.45:0/2312009366 learned_addr learned my addr 172.21.15.45:0/2312009366 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 client.?.objecter ms_handle_connect 0x7fdbac09fc40 2024-09-16T03:14:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3842098155 0 0) 0x7fdb9c003aa0 con 0x7fdbac09fc40 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): my global_id is 134178 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb8c004ec0 con 0x7fdbac09fc40 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 client.?.objecter ms_handle_connect 0x7fdbac13c010 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4052308632 0 0) 0x7fdb8c004ec0 con 0x7fdbac09fc40 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb9c003aa0 con 0x7fdbac09fc40 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdb9c0043b0 con 0x7fdbac09fc40 2024-09-16T03:14:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): dump: 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.643 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:44.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4042137885 0 0) 0x7fdbac09e5d0 con 0x7fdbac13c010 2024-09-16T03:14:44.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient(hunting): my global_id is 23089 2024-09-16T03:14:44.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb8c004ec0 con 0x7fdbac13c010 2024-09-16T03:14:44.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4280947853 0 0) 0x7fdb9c003aa0 con 0x7fdbac09fc40 2024-09-16T03:14:44.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 >> v1:172.21.15.45:6790/0 conn(0x7fdbac135950 legacy=0x7fdbac072ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:14:44.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 >> v1:172.21.15.46:6789/0 conn(0x7fdbac13c010 legacy=0x7fdbac1d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:14:44.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 monclient: found mon.a 2024-09-16T03:14:44.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:44.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac0735e0 con 0x7fdbac09fc40 2024-09-16T03:14:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient: _finish_auth 0 2024-09-16T03:14:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:14:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 5 monclient: authenticate success, global_id 134178 2024-09-16T03:14:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient: _renew_subs 2024-09-16T03:14:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbac09eb90 con 0x7fdbac09fc40 2024-09-16T03:14:44.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 client.134178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:44.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient: _renew_subs 2024-09-16T03:14:44.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbac09f540 con 0x7fdbac09fc40 2024-09-16T03:14:44.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 librados: init done 2024-09-16T03:14:44.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb9c004550 con 0x7fdbac09fc40 2024-09-16T03:14:44.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdb9c004960 con 0x7fdbac09fc40 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 monclient: dump: 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba9ffb640 4 set_mon_vals no callback set 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdb900065d0 con 0x7fdb90002870 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 client.134178.objecter ms_handle_connect 0x7fdb90002870 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 10 client.134178.objecter resend_mon_ops 2024-09-16T03:14:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.615+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fdb9c004b00 con 0x7fdbac09fc40 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951975408 0 0) 0x7fdb900065d0 con 0x7fdb90002870 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 monclient(hunting): my global_id is 134181 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb9c003aa0 con 0x7fdb90002870 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(391..391 src has 99..391) ==== 5374+0+0 (unknown 589942475 0 0) 0x7fdb9c04b900 con 0x7fdbac09fc40 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 client.134178.objecter ms_dispatch 0x7fdbac000ba0 osd_map(391..391 src has 99..391) 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 3 client.134178.objecter handle_osd_map got epochs [391,391] > 0 2024-09-16T03:14:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 3 client.134178.objecter handle_osd_map decoding full epoch 391 2024-09-16T03:14:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 20 client.134178.objecter dump_active .. 0 homeless 2024-09-16T03:14:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3328113696 0 0) 0x7fdb9c003aa0 con 0x7fdb90002870 2024-09-16T03:14:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb900065d0 con 0x7fdb90002870 2024-09-16T03:14:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdb940014c0 con 0x7fdb90002870 2024-09-16T03:14:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 monclient: dump: 2024-09-16T03:14:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.653 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdbabfff640 10 monclient: get_auth_request con 0x7fdb8c03e040 auth_method 0 2024-09-16T03:14:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2598734179 0 0) 0x7fdb900065d0 con 0x7fdb90002870 2024-09-16T03:14:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb9000a170 con 0x7fdb90002870 2024-09-16T03:14:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 client.134178.objecter ms_handle_connect 0x7fdb8c03e040 2024-09-16T03:14:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdb940014c0 con 0x7fdb90002870 2024-09-16T03:14:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 >> v1:172.21.15.45:6789/0 conn(0x7fdb90002870 legacy=0x7fdb90004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.619+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 >> v1:172.21.15.45:6789/0 conn(0x7fdb90002870 legacy=0x7fdb90004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdbb1f85640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:14:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdb90009e60 con 0x7fdb90009120 2024-09-16T03:14:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 client.134178.objecter ms_handle_connect 0x7fdb90009120 2024-09-16T03:14:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 client.134178.objecter resend_mon_ops 2024-09-16T03:14:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 196897888 0 0) 0x7fdb90009e60 con 0x7fdb90009120 2024-09-16T03:14:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 monclient(hunting): my global_id is 134184 2024-09-16T03:14:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb900065d0 con 0x7fdb90009120 2024-09-16T03:14:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2300261312 0 0) 0x7fdb900065d0 con 0x7fdb90009120 2024-09-16T03:14:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb90009e60 con 0x7fdb90009120 2024-09-16T03:14:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdb94004990 con 0x7fdb90009120 2024-09-16T03:14:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 monclient: dump: 2024-09-16T03:14:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.657 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.658 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1259913054 0 0) 0x7fdb90009e60 con 0x7fdb90009120 2024-09-16T03:14:44.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdb9000c400 con 0x7fdb90009120 2024-09-16T03:14:44.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4125 (unknown 0 0 471699139) 0x7fdb94004920 con 0x7fdb90009120 2024-09-16T03:14:44.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:14:44.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:44.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 >> v1:172.21.15.45:6789/0 conn(0x7fdb90009120 legacy=0x7fdb900095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.623+0000 7fdba3fff640 1 -- 172.21.15.45:0/2312009366 >> v1:172.21.15.45:6789/0 conn(0x7fdb90009120 legacy=0x7fdb900095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:44.659 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:14:44.659 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:44.659 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:44.659 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:14:44.659 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 44, 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 17851, 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: 1 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:14:44.660 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:44.661 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:44.662 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.662 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:14:44.662 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:14:44.662 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:44.662 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:44.662 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:44.662 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.663 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:14:44.664 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:14:44.665 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:14:44.665 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:14:44.665 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:14:44.665 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:44.665 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:14:44.665 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:44.665 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:44.665 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:14:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:14:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:14:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:14:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:14:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:14:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:14:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:14:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:14:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:14:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:14:44.676 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.676 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.676 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:14:44.676 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:14:44.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 10 librados: watch_flush enter 2024-09-16T03:14:44.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 10 librados: watch_flush exit 2024-09-16T03:14:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 20 client.134178.objecter shutdown clearing up homeless session... 2024-09-16T03:14:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 10 client.134178.objecter successfully canceled tick 2024-09-16T03:14:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 >> v1:172.21.15.46:6812/149360280 conn(0x7fdb8c03e040 legacy=0x7fdb8c040500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 10 monclient: shutdown 2024-09-16T03:14:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 >> v1:172.21.15.45:6789/0 conn(0x7fdbac09fc40 legacy=0x7fdbac0727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbabfff640 1 -- 172.21.15.45:0/2312009366 reap_dead start 2024-09-16T03:14:44.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbabfff640 1 -- 172.21.15.45:0/2312009366 reap_dead start 2024-09-16T03:14:44.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 shutdown_connections 2024-09-16T03:14:44.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 >> 172.21.15.45:0/2312009366 conn(0x7fdbac0a2bd0 msgr2=0x7fdbac13ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:44.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 shutdown_connections 2024-09-16T03:14:44.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 1 -- 172.21.15.45:0/2312009366 wait complete. 2024-09-16T03:14:44.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.627+0000 7fdbb1f85640 1 librados: shutdown 2024-09-16T03:14:44.680 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:14:44.681 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:14:44.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:14:44.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient: get_monmap_and_config 2024-09-16T03:14:44.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient: build_initial_monmap 2024-09-16T03:14:44.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient: monmap: 2024-09-16T03:14:44.792 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:44.793 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.793 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:14:44.793554+0000 2024-09-16T03:14:44.793 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:14:44.793554+0000 2024-09-16T03:14:44.793 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:44.793 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.793 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:44.793 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:14:44.793 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:14:44.793 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 1 Processor -- start 2024-09-16T03:14:44.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 1 -- start start 2024-09-16T03:14:44.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient: init 2024-09-16T03:14:44.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1c135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1c09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1c13c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c09f7e0 con 0x7f8e1c09fc40 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c09b160 con 0x7f8e1c135950 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c09b440 con 0x7f8e1c13c010 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient(hunting): authenticate will time out at 1575.442962s 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e21b0a640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f8e1c13c010 legacy=0x7f8e1c13e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e20b08640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8e1c09fc40 0x7f8e1c13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54564/0 (socket says 172.21.15.45:54564) 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e20b08640 1 -- 172.21.15.45:0/150685079 learned_addr learned my addr 172.21.15.45:0/150685079 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 -- 172.21.15.45:0/150685079 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944997970 0 0) 0x7f8e1c09f7e0 con 0x7f8e1c09fc40 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient(hunting): my global_id is 134187 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 -- 172.21.15.45:0/150685079 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e08003530 con 0x7f8e1c09fc40 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 -- 172.21.15.45:0/150685079 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1540183998 0 0) 0x7f8e08003530 con 0x7f8e1c09fc40 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 -- 172.21.15.45:0/150685079 >> v1:172.21.15.45:6790/0 conn(0x7f8e1c13c010 legacy=0x7f8e1c13e400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 -- 172.21.15.45:0/150685079 >> v1:172.21.15.46:6789/0 conn(0x7f8e1c135950 legacy=0x7f8e1c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: _finish_hunting 0 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 monclient: found mon.noname-a 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 -- 172.21.15.45:0/150685079 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1c136440 con 0x7f8e1c09fc40 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: _finish_auth 0 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: _check_auth_tickets 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 -- 172.21.15.45:0/150685079 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e180016a0 con 0x7f8e1c09fc40 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: dump: 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.798 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 5 monclient: authenticate success, global_id 134187 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 -- 172.21.15.45:0/150685079 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e18001840 con 0x7f8e1c09fc40 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 1 -- 172.21.15.45:0/150685079 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e18001cd0 con 0x7f8e1c09fc40 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 10 monclient: get_monmap_and_config success 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e2356d640 4 set_mon_vals no callback set 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:44.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.791+0000 7f8e13fff640 10 monclient: dump: 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: shutdown 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:44.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- 172.21.15.45:0/150685079 >> v1:172.21.15.45:6789/0 conn(0x7f8e1c09fc40 legacy=0x7f8e1c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- 172.21.15.45:0/150685079 shutdown_connections 2024-09-16T03:14:44.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- 172.21.15.45:0/150685079 >> 172.21.15.45:0/150685079 conn(0x7f8e1c0a2bd0 msgr2=0x7f8e1c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:44.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- 172.21.15.45:0/150685079 shutdown_connections 2024-09-16T03:14:44.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- 172.21.15.45:0/150685079 wait complete. 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: build_initial_monmap 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: monmap: 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:14:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 librados: starting msgr at 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 librados: starting objecter 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 Processor -- start 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- start start 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 librados: setting wanted keys 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 librados: calling monclient init 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: init 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: _reopen_session rank -1 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1c09fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:14:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1c135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1c13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e21309640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f8e1c09fc40 legacy=0x7f8e1c06e3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c09d2c0 con 0x7f8e1c135950 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c071250 con 0x7f8e1c13c010 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c071530 con 0x7f8e1c09fc40 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient(hunting): _renew_subs 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient(hunting): authenticate will time out at 1575.446810s 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e20b08640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8e1c135950 0x7f8e1c06eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54578/0 (socket says 172.21.15.45:54578) 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e20b08640 1 -- 172.21.15.45:0/3961913730 learned_addr learned my addr 172.21.15.45:0/3961913730 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 client.?.objecter ms_handle_connect 0x7f8e1c135950 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 698256876 0 0) 0x7f8e1c09d2c0 con 0x7f8e1c135950 2024-09-16T03:14:44.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): my global_id is 134190 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e00004ec0 con 0x7f8e1c135950 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 client.?.objecter ms_handle_connect 0x7f8e1c13c010 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3026781463 0 0) 0x7f8e00004ec0 con 0x7f8e1c135950 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e1c09d2c0 con 0x7f8e1c135950 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e18001b90 con 0x7f8e1c135950 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): dump: 2024-09-16T03:14:44.808 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 809053338 0 0) 0x7f8e1c071250 con 0x7f8e1c13c010 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient(hunting): my global_id is 23095 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e00004ec0 con 0x7f8e1c13c010 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1412647121 0 0) 0x7f8e1c09d2c0 con 0x7f8e1c135950 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.45:6790/0 conn(0x7f8e1c09fc40 legacy=0x7f8e1c06e3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8e1c13c010 legacy=0x7f8e1c06f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 monclient: found mon.a 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1c0702f0 con 0x7f8e1c135950 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient: _finish_auth 0 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:14:44.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 5 monclient: authenticate success, global_id 134190 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: _renew_subs 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1c13b540 con 0x7f8e1c135950 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 client.134190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: _renew_subs 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e1c1d3cd0 con 0x7f8e1c135950 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e2356d640 1 librados: init done 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e18004c40 con 0x7f8e1c135950 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:14:44.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e18004df0 con 0x7f8e1c135950 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e12ffd640 4 set_mon_vals no callback set 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.795+0000 7f8e11ffb640 10 monclient: dump: 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.812 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e2356d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8dfc0065d0 con 0x7f8dfc002870 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f8e18010e60 con 0x7f8e1c135950 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(391..391 src has 99..391) ==== 5374+0+0 (unknown 589942475 0 0) 0x7f8e1804b840 con 0x7f8e1c135950 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 client.134190.objecter ms_dispatch 0x7f8e1c000ba0 osd_map(391..391 src has 99..391) 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 3 client.134190.objecter handle_osd_map got epochs [391,391] > 0 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 3 client.134190.objecter handle_osd_map decoding full epoch 391 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 20 client.134190.objecter dump_active .. 0 homeless 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 client.134190.objecter ms_handle_connect 0x7f8dfc002870 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 client.134190.objecter resend_mon_ops 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021269084 0 0) 0x7f8dfc0065d0 con 0x7f8dfc002870 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 monclient(hunting): my global_id is 23098 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e1c09d2c0 con 0x7f8dfc002870 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e21b0a640 10 monclient: get_auth_request con 0x7f8e0003e000 auth_method 0 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 999289698 0 0) 0x7f8e1c09d2c0 con 0x7f8dfc002870 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dfc0065d0 con 0x7f8dfc002870 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e0c0017c0 con 0x7f8dfc002870 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 monclient: dump: 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.814 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 client.134190.objecter ms_handle_connect 0x7f8e0003e000 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3600417576 0 0) 0x7f8dfc0065d0 con 0x7f8dfc002870 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8dfc00a170 con 0x7f8dfc002870 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8e0c001a00 con 0x7f8dfc002870 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:14:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:14:44.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8dfc002870 legacy=0x7f8dfc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.799+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8dfc002870 legacy=0x7f8dfc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:44.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e2356d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:14:44.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8dfc009e80 con 0x7f8dfc009140 2024-09-16T03:14:44.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 client.134190.objecter ms_handle_connect 0x7f8dfc009140 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 client.134190.objecter resend_mon_ops 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348758072 0 0) 0x7f8dfc009e80 con 0x7f8dfc009140 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient(hunting): my global_id is 23101 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dfc0065d0 con 0x7f8dfc009140 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e21309640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8dfc009140 legacy=0x7f8dfc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e21309640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8dfc009140 legacy=0x7f8dfc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e21309640 1 --1- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8dfc009140 0x7f8dfc0095d0 :-1 s=OPENED pgs=8125 cs=1 l=1).handle_message read tag failed 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e21309640 1 --1- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8dfc009140 0x7f8dfc0095d0 :-1 s=OPENED pgs=8125 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8dfc009140 legacy=0x7f8dfc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8dfc009140 legacy=0x7f8dfc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8dfc009e80 con 0x7f8e000490f0 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 client.134190.objecter ms_handle_connect 0x7f8e000490f0 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 client.134190.objecter resend_mon_ops 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1042234694 0 0) 0x7f8dfc009e80 con 0x7f8e000490f0 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient(hunting): my global_id is 23104 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e1c09d2c0 con 0x7f8e000490f0 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3607696759 0 0) 0x7f8e1c09d2c0 con 0x7f8e000490f0 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dfc009e80 con 0x7f8e000490f0 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8e0c0037a0 con 0x7f8e000490f0 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:14:44.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient: dump: 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2090356262 0 0) 0x7f8dfc009e80 con 0x7f8e000490f0 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8e0c001a00 con 0x7f8e000490f0 2024-09-16T03:14:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4291 (unknown 0 0 4045458183) 0x7f8e0c0057d0 con 0x7f8e000490f0 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8e000490f0 legacy=0x7f8e00047aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e21b0a640 1 -- 172.21.15.45:0/3961913730 reap_dead start 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.807+0000 7f8e11ffb640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6789/0 conn(0x7f8e000490f0 legacy=0x7f8e00047aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:14:44.822 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 44, 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 606613, 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: 1 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:14:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:14:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:14:44.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.828 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:14:44.829 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:14:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.831 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:44.832 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:14:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:14:44.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:14:44.834 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:14:44.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:14:44.834 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:14:44.834 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:14:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 10 librados: watch_flush enter 2024-09-16T03:14:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 10 librados: watch_flush exit 2024-09-16T03:14:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 20 client.134190.objecter shutdown clearing up homeless session... 2024-09-16T03:14:44.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 10 client.134190.objecter successfully canceled tick 2024-09-16T03:14:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.46:6812/149360280 conn(0x7f8e0003e000 legacy=0x7f8e000404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 10 monclient: shutdown 2024-09-16T03:14:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:14:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 >> v1:172.21.15.45:6789/0 conn(0x7f8e1c135950 legacy=0x7f8e1c06eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:14:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 shutdown_connections 2024-09-16T03:14:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 >> 172.21.15.45:0/3961913730 conn(0x7f8e1c0a2bd0 msgr2=0x7f8e1c0a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:14:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 shutdown_connections 2024-09-16T03:14:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 1 -- 172.21.15.45:0/3961913730 wait complete. 2024-09-16T03:14:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:14:44.811+0000 7f8e2356d640 1 librados: shutdown 2024-09-16T03:14:44.836 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:14:44.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:15:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (26065 ms) 2024-09-16T03:15:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-16T03:15:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22901 ms) 2024-09-16T03:15:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-16T03:15:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3042 ms) 2024-09-16T03:15:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-16T03:15:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3024 ms) 2024-09-16T03:15:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-16T03:15:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2933 ms) 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3009 ms) 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4044 ms) 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39967 ms) 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (28127 ms) 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-16T03:15:01.823 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4041 ms) 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (2188 ms) 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25092 ms) 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15069 ms) 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-16T03:15:01.824 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (4852 ms) 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4035 ms) 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: done waiting 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25081 ms) 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9601 ms) 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-16T03:15:01.825 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (24137 ms) 2024-09-16T03:15:01.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-16T03:15:01.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:15:01.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8649 ms) 2024-09-16T03:15:01.826 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-16T03:15:58.907 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-09-16T03:15:58.906+0000 7f63552d7640 -1 snap_mapper.add_oid found existing snaps mapped on 105:2b9fb66f:test-rados-api-smithi046-28678-97::foo:21, removing 2024-09-16T03:15:58.908 INFO:tasks.ceph.osd.4.smithi046.stderr:2024-09-16T03:15:58.901+0000 7f9674062640 -1 snap_mapper.add_oid found existing snaps mapped on 105:2b9fb66f:test-rados-api-smithi046-28678-97::foo:21, removing 2024-09-16T03:16:14.838 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:16:14.838 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:16:14.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:16:14.841 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:16:14.841 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:16:14.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:16:14.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient: get_monmap_and_config 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient: build_initial_monmap 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient: monmap: 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:14.968587+0000 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:14.968587+0000 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:14.967 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:14.968 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 1 Processor -- start 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 1 -- start start 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient: init 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient(hunting): picked mon.noname-c con 0x7f73d0137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient(hunting): picked mon.noname-a con 0x7f73d0138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient(hunting): picked mon.noname-b con 0x7f73d013e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d00a4f90 con 0x7f73d0138100 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d00a4cb0 con 0x7f73d0137d20 2024-09-16T03:16:14.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d009dc70 con 0x7f73d013e2a0 2024-09-16T03:16:14.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:14.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73d5959640 10 monclient(hunting): authenticate will time out at 1665.617608s 2024-09-16T03:16:14.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73ce7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f73d0138100 0x7f73d013cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39392/0 (socket says 172.21.15.45:39392) 2024-09-16T03:16:14.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.966+0000 7f73ce7fc640 1 -- 172.21.15.45:0/1136559817 learned_addr learned my addr 172.21.15.45:0/1136559817 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:14.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 -- 172.21.15.45:0/1136559817 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1125331038 0 0) 0x7f73d00a4f90 con 0x7f73d0138100 2024-09-16T03:16:14.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:16:14.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:14.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:14.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient(hunting): my global_id is 134193 2024-09-16T03:16:14.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 -- 172.21.15.45:0/1136559817 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73bc003530 con 0x7f73d0138100 2024-09-16T03:16:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 -- 172.21.15.45:0/1136559817 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2903745037 0 0) 0x7f73bc003530 con 0x7f73d0138100 2024-09-16T03:16:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 -- 172.21.15.45:0/1136559817 >> v1:172.21.15.45:6790/0 conn(0x7f73d013e2a0 legacy=0x7f73d0140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 -- 172.21.15.45:0/1136559817 >> v1:172.21.15.46:6789/0 conn(0x7f73d0137d20 legacy=0x7f73d00a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: _finish_hunting 0 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 monclient: found mon.noname-a 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 -- 172.21.15.45:0/1136559817 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d009cc50 con 0x7f73d0138100 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: _finish_auth 0 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: _check_auth_tickets 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 -- 172.21.15.45:0/1136559817 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f73b80016a0 con 0x7f73d0138100 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: dump: 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73d5959640 5 monclient: authenticate success, global_id 134193 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73d5959640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 -- 172.21.15.45:0/1136559817 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73b8001840 con 0x7f73d0138100 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 1 -- 172.21.15.45:0/1136559817 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f73b8001c90 con 0x7f73d0138100 2024-09-16T03:16:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73d5959640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73d5959640 4 set_mon_vals no callback set 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.986+0000 7f73cdffb640 10 monclient: dump: 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:14.994 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: shutdown 2024-09-16T03:16:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- 172.21.15.45:0/1136559817 >> v1:172.21.15.45:6789/0 conn(0x7f73d0138100 legacy=0x7f73d013cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- 172.21.15.45:0/1136559817 shutdown_connections 2024-09-16T03:16:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- 172.21.15.45:0/1136559817 >> 172.21.15.45:0/1136559817 conn(0x7f73d00aa110 msgr2=0x7f73d00aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- 172.21.15.45:0/1136559817 shutdown_connections 2024-09-16T03:16:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- 172.21.15.45:0/1136559817 wait complete. 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: build_initial_monmap 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: monmap: 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 librados: starting msgr at 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 librados: starting objecter 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 Processor -- start 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- start start 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 librados: setting wanted keys 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 librados: calling monclient init 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: init 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient(hunting): picked mon.noname-c con 0x7f73d0137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient(hunting): picked mon.noname-a con 0x7f73d0138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient(hunting): picked mon.noname-b con 0x7f73d013e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d013c080 con 0x7f73d0138100 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d0139770 con 0x7f73d013e2a0 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d0139d20 con 0x7f73d0137d20 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient(hunting): authenticate will time out at 1665.641837s 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73ce7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f73d0138100 0x7f73d0139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39404/0 (socket says 172.21.15.45:39404) 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73ce7fc640 1 -- 172.21.15.45:0/2421072951 learned_addr learned my addr 172.21.15.45:0/2421072951 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 client.?.objecter ms_handle_connect 0x7f73d0137d20 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 client.?.objecter ms_handle_connect 0x7f73d0138100 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1599562485 0 0) 0x7f73d013c080 con 0x7f73d0138100 2024-09-16T03:16:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient(hunting): my global_id is 134196 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73b0004ec0 con 0x7f73d0138100 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 client.?.objecter ms_handle_connect 0x7f73d013e2a0 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 194836737 0 0) 0x7f73b0004ec0 con 0x7f73d0138100 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73d013c080 con 0x7f73d0138100 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f73b8001bd0 con 0x7f73d0138100 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:15.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient(hunting): dump: 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 761523778 0 0) 0x7f73d013c080 con 0x7f73d0138100 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 >> v1:172.21.15.45:6790/0 conn(0x7f73d0137d20 legacy=0x7f73d0138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 >> v1:172.21.15.46:6789/0 conn(0x7f73d013e2a0 legacy=0x7f73d01d4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 monclient: found mon.a 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d013bda0 con 0x7f73d0138100 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient: _finish_auth 0 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 5 monclient: authenticate success, global_id 134196 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: _renew_subs 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73d013cea0 con 0x7f73d0138100 2024-09-16T03:16:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 client.134196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: _renew_subs 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f73d00a0e50 con 0x7f73d0138100 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 librados: init done 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73b80017a0 con 0x7f73d0138100 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f73b8004da0 con 0x7f73d0138100 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 10 monclient: dump: 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.004 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73cd7fa640 4 set_mon_vals no callback set 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f73b8004f40 con 0x7f73d0138100 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.990+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73b4005090 con 0x7f73d0138100 2024-09-16T03:16:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.994+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(455..455 src has 99..455) ==== 5367+0+0 (unknown 3166234270 0 0) 0x7f73b804b880 con 0x7f73d0138100 2024-09-16T03:16:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.994+0000 7f73c7fff640 10 client.134196.objecter ms_dispatch 0x7f73d0000ba0 osd_map(455..455 src has 99..455) 2024-09-16T03:16:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.994+0000 7f73c7fff640 3 client.134196.objecter handle_osd_map got epochs [455,455] > 0 2024-09-16T03:16:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.994+0000 7f73c7fff640 3 client.134196.objecter handle_osd_map decoding full epoch 455 2024-09-16T03:16:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.994+0000 7f73c7fff640 20 client.134196.objecter dump_active .. 0 homeless 2024-09-16T03:16:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.994+0000 7f73ceffd640 10 monclient: get_auth_request con 0x7f73b003e010 auth_method 0 2024-09-16T03:16:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.994+0000 7f73c7fff640 10 client.134196.objecter ms_handle_connect 0x7f73b003e010 2024-09-16T03:16:15.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.998+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f73b801ae90 con 0x7f73d0138100 2024-09-16T03:16:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.998+0000 7f73c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:14.998+0000 7f73c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:15.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.230+0000 7f73d5959640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:16:15.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.230+0000 7f73d5959640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:15.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.230+0000 7f73d5959640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.230+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f73b4005380 con 0x7f73d0138100 2024-09-16T03:16:15.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.230+0000 7f73c7fff640 1 -- 172.21.15.45:0/2421072951 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2705386306) 0x7f73b801fdd0 con 0x7f73d0138100 2024-09-16T03:16:15.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.230+0000 7f73c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:15.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.230+0000 7f73c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:15.233 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 10 librados: watch_flush enter 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 10 librados: watch_flush exit 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 20 client.134196.objecter shutdown clearing up homeless session... 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 10 client.134196.objecter successfully canceled tick 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 >> v1:172.21.15.46:6812/149360280 conn(0x7f73b003e010 legacy=0x7f73b00404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 10 monclient: shutdown 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 >> v1:172.21.15.45:6789/0 conn(0x7f73d0138100 legacy=0x7f73d0139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 shutdown_connections 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 >> 172.21.15.45:0/2421072951 conn(0x7f73d00aa110 msgr2=0x7f73d00a3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:15.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 shutdown_connections 2024-09-16T03:16:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 1 -- 172.21.15.45:0/2421072951 wait complete. 2024-09-16T03:16:15.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.234+0000 7f73d5959640 1 librados: shutdown 2024-09-16T03:16:15.251 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:16:15.251 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:16:15.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:16:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.370+0000 7f0fce6ce640 10 monclient: get_monmap_and_config 2024-09-16T03:16:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.370+0000 7f0fce6ce640 10 monclient: build_initial_monmap 2024-09-16T03:16:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.370+0000 7f0fce6ce640 10 monclient: monmap: 2024-09-16T03:16:15.373 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:15.375240+0000 2024-09-16T03:16:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:15.375240+0000 2024-09-16T03:16:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:15.374 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:15.374 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.370+0000 7f0fce6ce640 1 Processor -- start 2024-09-16T03:16:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.370+0000 7f0fce6ce640 1 -- start start 2024-09-16T03:16:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.370+0000 7f0fce6ce640 10 monclient: init 2024-09-16T03:16:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:16:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc80a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc80a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc8073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc809d9e0 con 0x7f0fc8073d40 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc809dc20 con 0x7f0fc80a4d20 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc8073360 con 0x7f0fc80a5810 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 10 monclient(hunting): authenticate will time out at 1666.024814s 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fccc6b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0fc8073d40 0x7f0fc8140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39408/0 (socket says 172.21.15.45:39408) 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fccc6b640 1 -- 172.21.15.45:0/1344834074 learned_addr learned my addr 172.21.15.45:0/1344834074 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1291354878 0 0) 0x7f0fc809d9e0 con 0x7f0fc8073d40 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient(hunting): my global_id is 134199 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fb0003530 con 0x7f0fc8073d40 2024-09-16T03:16:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2578687485 0 0) 0x7f0fc809dc20 con 0x7f0fc80a4d20 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient(hunting): my global_id is 23107 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fc809d9e0 con 0x7f0fc80a4d20 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 185260395 0 0) 0x7f0fc809d9e0 con 0x7f0fc80a4d20 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 >> v1:172.21.15.45:6790/0 conn(0x7f0fc80a5810 legacy=0x7f0fc813cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 >> v1:172.21.15.45:6789/0 conn(0x7f0fc8073d40 legacy=0x7f0fc8140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 monclient: found mon.noname-c 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.46:6789/0 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc809cf30 con 0x7f0fc80a4d20 2024-09-16T03:16:15.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0fb8001680 con 0x7f0fc80a4d20 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 5 monclient: authenticate success, global_id 23107 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: dump: 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 <== mon.1 v1:172.21.15.46:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0fb80018a0 con 0x7f0fc80a4d20 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 4 set_mon_vals no callback set 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 1 -- 172.21.15.45:0/1344834074 <== mon.1 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0fb8001cb0 con 0x7f0fc80a4d20 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fc6ffd640 10 monclient: dump: 2024-09-16T03:16:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.382 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 10 monclient: shutdown 2024-09-16T03:16:15.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:15.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 1 -- 172.21.15.45:0/1344834074 >> v1:172.21.15.46:6789/0 conn(0x7f0fc80a4d20 legacy=0x7f0fc80a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 1 -- 172.21.15.45:0/1344834074 shutdown_connections 2024-09-16T03:16:15.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 1 -- 172.21.15.45:0/1344834074 >> 172.21.15.45:0/1344834074 conn(0x7f0fc80aa720 msgr2=0x7f0fc80aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:15.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 1 -- 172.21.15.45:0/1344834074 shutdown_connections 2024-09-16T03:16:15.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.374+0000 7f0fce6ce640 1 -- 172.21.15.45:0/1344834074 wait complete. 2024-09-16T03:16:15.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient: build_initial_monmap 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient: monmap: 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 librados: starting msgr at 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 librados: starting objecter 2024-09-16T03:16:15.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 Processor -- start 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 -- start start 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 librados: setting wanted keys 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 librados: calling monclient init 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient: init 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc8073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc80a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc80a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc813f6e0 con 0x7f0fc80a5810 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc8140a20 con 0x7f0fc80a4d20 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc813d550 con 0x7f0fc8073d40 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:15.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient(hunting): authenticate will time out at 1666.029424s 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc7fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0fc8073d40 0x7f0fc813c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37384/0 (socket says 172.21.15.45:37384) 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc7fff640 1 -- 172.21.15.45:0/488715609 learned_addr learned my addr 172.21.15.45:0/488715609 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 client.?.objecter ms_handle_connect 0x7f0fc8073d40 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 client.?.objecter ms_handle_connect 0x7f0fc80a5810 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417321591 0 0) 0x7f0fc813d550 con 0x7f0fc8073d40 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): my global_id is 144101 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fb4004ec0 con 0x7f0fc8073d40 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3336152408 0 0) 0x7f0fc813f6e0 con 0x7f0fc80a5810 2024-09-16T03:16:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): my global_id is 134202 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fc813d550 con 0x7f0fc80a5810 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 client.?.objecter ms_handle_connect 0x7f0fc80a4d20 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 552399471 0 0) 0x7f0fb4004ec0 con 0x7f0fc8073d40 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fc813f6e0 con 0x7f0fc8073d40 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0fb8001d30 con 0x7f0fc8073d40 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): dump: 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2753453937 0 0) 0x7f0fc813d550 con 0x7f0fc80a5810 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fb4004ec0 con 0x7f0fc80a5810 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0fc00014c0 con 0x7f0fc80a5810 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:15.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): dump: 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3695872191 0 0) 0x7f0fc813f6e0 con 0x7f0fc8073d40 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 >> v1:172.21.15.46:6789/0 conn(0x7f0fc80a4d20 legacy=0x7f0fc813ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 >> v1:172.21.15.45:6789/0 conn(0x7f0fc80a5810 legacy=0x7f0fc81d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 monclient: found mon.c 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc813e1f0 con 0x7f0fc8073d40 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient: _finish_auth 0 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 5 monclient: authenticate success, global_id 144101 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient: _renew_subs 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc8140550 con 0x7f0fc8073d40 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 client.144101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient: _renew_subs 2024-09-16T03:16:15.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0fc813efb0 con 0x7f0fc8073d40 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 librados: init done 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0fb8003ab0 con 0x7f0fc8073d40 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0fb8003e40 con 0x7f0fc8073d40 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient: dump: 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.396 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc67fc640 4 set_mon_vals no callback set 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f0fb8004060 con 0x7f0fc8073d40 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0fa80065d0 con 0x7f0fa8002870 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(455..455 src has 99..455) ==== 5367+0+0 (unknown 3166234270 0 0) 0x7f0fb804b8d0 con 0x7f0fc8073d40 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 client.144101.objecter ms_dispatch 0x7f0fc8000ba0 osd_map(455..455 src has 99..455) 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 3 client.144101.objecter handle_osd_map got epochs [455,455] > 0 2024-09-16T03:16:15.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 3 client.144101.objecter handle_osd_map decoding full epoch 455 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 20 client.144101.objecter dump_active .. 0 homeless 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 client.144101.objecter ms_handle_connect 0x7f0fa8002870 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 client.144101.objecter resend_mon_ops 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3131531381 0 0) 0x7f0fa80065d0 con 0x7f0fa8002870 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 10 monclient(hunting): my global_id is 134205 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fc813f6e0 con 0x7f0fa8002870 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.378+0000 7f0fccc6b640 10 monclient: get_auth_request con 0x7f0fb403e1d0 auth_method 0 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3020062218 0 0) 0x7f0fc813f6e0 con 0x7f0fa8002870 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fa80065d0 con 0x7f0fa8002870 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0fbc001750 con 0x7f0fa8002870 2024-09-16T03:16:15.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 10 monclient: dump: 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2504210440 0 0) 0x7f0fa80065d0 con 0x7f0fa8002870 2024-09-16T03:16:15.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fa800a170 con 0x7f0fa8002870 2024-09-16T03:16:15.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 10 client.144101.objecter ms_handle_connect 0x7f0fb403e1d0 2024-09-16T03:16:15.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0fbc001990 con 0x7f0fa8002870 2024-09-16T03:16:15.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:15.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 >> v1:172.21.15.45:6789/0 conn(0x7f0fa8002870 legacy=0x7f0fa8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.382+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 >> v1:172.21.15.45:6789/0 conn(0x7f0fa8002870 legacy=0x7f0fa8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:15.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:16:15.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0fa8009e80 con 0x7f0fa8009140 2024-09-16T03:16:15.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 client.144101.objecter ms_handle_connect 0x7f0fa8009140 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 client.144101.objecter resend_mon_ops 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079590967 0 0) 0x7f0fa8009e80 con 0x7f0fa8009140 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 monclient(hunting): my global_id is 134208 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fa80065d0 con 0x7f0fa8009140 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1128509120 0 0) 0x7f0fa80065d0 con 0x7f0fa8009140 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fa8009e80 con 0x7f0fa8009140 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0fbc0037a0 con 0x7f0fa8009140 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:15.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 monclient: dump: 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2320532694 0 0) 0x7f0fa8009e80 con 0x7f0fa8009140 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0fa800c420 con 0x7f0fa8009140 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4137 (unknown 0 0 302466740) 0x7f0fbc003730 con 0x7f0fa8009140 2024-09-16T03:16:15.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 >> v1:172.21.15.45:6789/0 conn(0x7f0fa8009140 legacy=0x7f0fa80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fc4ff9640 1 -- 172.21.15.45:0/488715609 >> v1:172.21.15.45:6789/0 conn(0x7f0fa8009140 legacy=0x7f0fa80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 46, 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 108614, 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:16:15.404 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:15.405 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:15.406 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:16:15.407 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:15.408 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:15.410 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.411 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.412 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.413 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.414 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:16:15.415 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:16:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 10 librados: watch_flush enter 2024-09-16T03:16:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 10 librados: watch_flush exit 2024-09-16T03:16:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 20 client.144101.objecter shutdown clearing up homeless session... 2024-09-16T03:16:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 10 client.144101.objecter successfully canceled tick 2024-09-16T03:16:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 >> v1:172.21.15.46:6812/149360280 conn(0x7f0fb403e1d0 legacy=0x7f0fb4040690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 10 monclient: shutdown 2024-09-16T03:16:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 >> v1:172.21.15.45:6790/0 conn(0x7f0fc8073d40 legacy=0x7f0fc813c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.386+0000 7f0fccc6b640 1 -- 172.21.15.45:0/488715609 reap_dead start 2024-09-16T03:16:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.390+0000 7f0fccc6b640 1 -- 172.21.15.45:0/488715609 reap_dead start 2024-09-16T03:16:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.390+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 shutdown_connections 2024-09-16T03:16:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.390+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 >> 172.21.15.45:0/488715609 conn(0x7f0fc80aa720 msgr2=0x7f0fc80a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.390+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 shutdown_connections 2024-09-16T03:16:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.390+0000 7f0fce6ce640 1 -- 172.21.15.45:0/488715609 wait complete. 2024-09-16T03:16:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.390+0000 7f0fce6ce640 1 librados: shutdown 2024-09-16T03:16:15.418 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:16:15.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:16:15.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:16:15.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient: get_monmap_and_config 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient: build_initial_monmap 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient: monmap: 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:15.537249+0000 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:15.537249+0000 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:15.536 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 1 Processor -- start 2024-09-16T03:16:15.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 1 -- start start 2024-09-16T03:16:15.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient: init 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient(hunting): picked mon.noname-b con 0x7fc2ec137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient(hunting): picked mon.noname-c con 0x7fc2ec138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient(hunting): picked mon.noname-a con 0x7fc2ec13e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2ec0a4f90 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2ec0a4cb0 con 0x7fc2ec138100 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2ec09dc70 con 0x7fc2ec137d20 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 10 monclient(hunting): authenticate will time out at 1666.187019s 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ebfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc2ec13e2a0 0x7fc2ec140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39448/0 (socket says 172.21.15.45:39448) 2024-09-16T03:16:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ebfff640 1 -- 172.21.15.45:0/2237465753 learned_addr learned my addr 172.21.15.45:0/2237465753 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:15.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2eb7fe640 1 --1- 172.21.15.45:0/2237465753 >> v1:172.21.15.45:6790/0 conn(0x7fc2ec137d20 0x7fc2ec0a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37394/0 (socket says 172.21.15.45:37394) 2024-09-16T03:16:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 424501531 0 0) 0x7fc2ec0a4f90 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient(hunting): my global_id is 134211 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d4003530 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1257498857 0 0) 0x7fc2ec09dc70 con 0x7fc2ec137d20 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient(hunting): my global_id is 144104 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2ec0a4f90 con 0x7fc2ec137d20 2024-09-16T03:16:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3887843434 0 0) 0x7fc2d4003530 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 >> v1:172.21.15.45:6790/0 conn(0x7fc2ec137d20 legacy=0x7fc2ec0a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 >> v1:172.21.15.46:6789/0 conn(0x7fc2ec138100 legacy=0x7fc2ec13cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:16:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 monclient: found mon.noname-a 2024-09-16T03:16:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ec09cc50 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient: _finish_auth 0 2024-09-16T03:16:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc2e40016a0 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2f1fb1640 5 monclient: authenticate success, global_id 134211 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.534+0000 7fc2ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2ea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2ea7fc640 10 monclient: dump: 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.551 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2e40018c0 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2ea7fc640 1 -- 172.21.15.45:0/2237465753 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc2e4001d50 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2ea7fc640 10 monclient: dump: 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:15.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 4 set_mon_vals no callback set 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: shutdown 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/2237465753 >> v1:172.21.15.45:6789/0 conn(0x7fc2ec13e2a0 legacy=0x7fc2ec140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/2237465753 shutdown_connections 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/2237465753 >> 172.21.15.45:0/2237465753 conn(0x7fc2ec0aa110 msgr2=0x7fc2ec0aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/2237465753 shutdown_connections 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/2237465753 wait complete. 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: build_initial_monmap 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: monmap: 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:15.554 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 librados: starting msgr at 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 librados: starting objecter 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 Processor -- start 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- start start 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 librados: setting wanted keys 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 librados: calling monclient init 2024-09-16T03:16:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: init 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient(hunting): picked mon.noname-b con 0x7fc2ec137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient(hunting): picked mon.noname-a con 0x7fc2ec138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient(hunting): picked mon.noname-c con 0x7fc2ec13e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2ec13c230 con 0x7fc2ec138100 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2ec139770 con 0x7fc2ec137d20 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2ec0a1300 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient(hunting): authenticate will time out at 1666.190233s 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2eaffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc2ec138100 0x7fc2ec139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39464/0 (socket says 172.21.15.45:39464) 2024-09-16T03:16:15.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2eaffd640 1 -- 172.21.15.45:0/3011727699 learned_addr learned my addr 172.21.15.45:0/3011727699 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:15.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 client.?.objecter ms_handle_connect 0x7fc2ec138100 2024-09-16T03:16:15.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 client.?.objecter ms_handle_connect 0x7fc2ec13e2a0 2024-09-16T03:16:15.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3199957627 0 0) 0x7fc2e0001600 con 0x7fc2ec138100 2024-09-16T03:16:15.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): my global_id is 134214 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d8004ec0 con 0x7fc2ec138100 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3016417836 0 0) 0x7fc2ec0a1300 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): my global_id is 144107 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2e0001600 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 client.?.objecter ms_handle_connect 0x7fc2ec137d20 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3786998830 0 0) 0x7fc2d8004ec0 con 0x7fc2ec138100 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2ec0a1300 con 0x7fc2ec138100 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc2e0001ef0 con 0x7fc2ec138100 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:15.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): dump: 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 293201137 0 0) 0x7fc2e0001600 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2d8004ec0 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157197466 0 0) 0x7fc2ec139770 con 0x7fc2ec137d20 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): my global_id is 23116 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2e0001600 con 0x7fc2ec137d20 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc2e4001a70 con 0x7fc2ec13e2a0 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient(hunting): dump: 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3711174389 0 0) 0x7fc2ec0a1300 con 0x7fc2ec138100 2024-09-16T03:16:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 >> v1:172.21.15.45:6790/0 conn(0x7fc2ec13e2a0 legacy=0x7fc2ec1d4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 >> v1:172.21.15.46:6789/0 conn(0x7fc2ec137d20 legacy=0x7fc2ec138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 monclient: found mon.a 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ec13b570 con 0x7fc2ec138100 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient: _finish_auth 0 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 5 monclient: authenticate success, global_id 134214 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: _renew_subs 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc2ec13a870 con 0x7fc2ec138100 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 client.134214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: _renew_subs 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc2ec13d740 con 0x7fc2ec138100 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 librados: init done 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2e0003c50 con 0x7fc2ec138100 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc2e0004020 con 0x7fc2ec138100 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2e9ffb640 4 set_mon_vals no callback set 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2d3fff640 10 monclient: dump: 2024-09-16T03:16:15.563 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.538+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc2cc0065d0 con 0x7fc2cc002870 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 client.134214.objecter ms_handle_connect 0x7fc2cc002870 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 client.134214.objecter resend_mon_ops 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fc2e00041c0 con 0x7fc2ec138100 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(455..455 src has 99..455) ==== 5367+0+0 (unknown 3166234270 0 0) 0x7fc2e004bcb0 con 0x7fc2ec138100 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 client.134214.objecter ms_dispatch 0x7fc2ec000ba0 osd_map(455..455 src has 99..455) 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 3 client.134214.objecter handle_osd_map got epochs [455,455] > 0 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 3 client.134214.objecter handle_osd_map decoding full epoch 455 2024-09-16T03:16:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 20 client.134214.objecter dump_active .. 0 homeless 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 365208517 0 0) 0x7fc2cc0065d0 con 0x7fc2cc002870 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 monclient(hunting): my global_id is 144110 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2ec0a1300 con 0x7fc2cc002870 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3962212817 0 0) 0x7fc2ec0a1300 con 0x7fc2cc002870 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2cc0065d0 con 0x7fc2cc002870 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2ebfff640 10 monclient: get_auth_request con 0x7fc2d803e2d0 auth_method 0 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc2dc0016f0 con 0x7fc2cc002870 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 monclient: dump: 2024-09-16T03:16:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1241511685 0 0) 0x7fc2cc0065d0 con 0x7fc2cc002870 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2cc00a170 con 0x7fc2cc002870 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 client.134214.objecter ms_handle_connect 0x7fc2d803e2d0 2024-09-16T03:16:15.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc2dc001950 con 0x7fc2cc002870 2024-09-16T03:16:15.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:15.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 >> v1:172.21.15.45:6790/0 conn(0x7fc2cc002870 legacy=0x7fc2cc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.542+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 >> v1:172.21.15.45:6790/0 conn(0x7fc2cc002870 legacy=0x7fc2cc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:15.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.546+0000 7fc2f1fb1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:16:15.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.546+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc2cc009e60 con 0x7fc2cc009120 2024-09-16T03:16:15.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.546+0000 7fc2d3fff640 10 client.134214.objecter ms_handle_connect 0x7fc2cc009120 2024-09-16T03:16:15.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.546+0000 7fc2d3fff640 10 client.134214.objecter resend_mon_ops 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229465768 0 0) 0x7fc2cc009e60 con 0x7fc2cc009120 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 10 monclient(hunting): my global_id is 144113 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2cc0065d0 con 0x7fc2cc009120 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2484353081 0 0) 0x7fc2cc0065d0 con 0x7fc2cc009120 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2cc009e60 con 0x7fc2cc009120 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc2dc004990 con 0x7fc2cc009120 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 10 monclient: dump: 2024-09-16T03:16:15.570 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3965740147 0 0) 0x7fc2cc009e60 con 0x7fc2cc009120 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2cc00c400 con 0x7fc2cc009120 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3965 (unknown 0 0 1022690819) 0x7fc2dc004920 con 0x7fc2cc009120 2024-09-16T03:16:15.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 >> v1:172.21.15.45:6790/0 conn(0x7fc2cc009120 legacy=0x7fc2cc0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2d3fff640 1 -- 172.21.15.45:0/3011727699 >> v1:172.21.15.45:6790/0 conn(0x7fc2cc009120 legacy=0x7fc2cc0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 46, 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 623, 2024-09-16T03:16:15.572 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:16:15.573 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:16:15.573 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:16:15.573 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:16:15.573 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:15.574 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:15.575 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:16:15.576 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:16:15.577 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.578 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:16:15.579 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:16:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:16:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:16:15.584 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:16:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 10 librados: watch_flush enter 2024-09-16T03:16:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 10 librados: watch_flush exit 2024-09-16T03:16:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 20 client.134214.objecter shutdown clearing up homeless session... 2024-09-16T03:16:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 10 client.134214.objecter successfully canceled tick 2024-09-16T03:16:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 >> v1:172.21.15.46:6812/149360280 conn(0x7fc2d803e2d0 legacy=0x7fc2d8040790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 10 monclient: shutdown 2024-09-16T03:16:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 >> v1:172.21.15.45:6789/0 conn(0x7fc2ec138100 legacy=0x7fc2ec139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2ebfff640 1 -- 172.21.15.45:0/3011727699 reap_dead start 2024-09-16T03:16:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2ebfff640 1 -- 172.21.15.45:0/3011727699 reap_dead start 2024-09-16T03:16:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 shutdown_connections 2024-09-16T03:16:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 >> 172.21.15.45:0/3011727699 conn(0x7fc2ec0aa110 msgr2=0x7fc2ec0731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 shutdown_connections 2024-09-16T03:16:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 1 -- 172.21.15.45:0/3011727699 wait complete. 2024-09-16T03:16:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.550+0000 7fc2f1fb1640 1 librados: shutdown 2024-09-16T03:16:15.587 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:16:15.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:16:15.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:16:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient: get_monmap_and_config 2024-09-16T03:16:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient: build_initial_monmap 2024-09-16T03:16:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient: monmap: 2024-09-16T03:16:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:15.700850+0000 2024-09-16T03:16:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:15.700850+0000 2024-09-16T03:16:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:15.700 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 Processor -- start 2024-09-16T03:16:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 -- start start 2024-09-16T03:16:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient: init 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient(hunting): picked mon.noname-b con 0x7f3ad0135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient(hunting): picked mon.noname-c con 0x7f3ad009fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient(hunting): picked mon.noname-a con 0x7f3ad013c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ad009f7e0 con 0x7f3ad013c010 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ad009b160 con 0x7f3ad009fc40 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ad009b440 con 0x7f3ad0135950 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient(hunting): authenticate will time out at 1666.350262s 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad48b5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3ad013c010 0x7f3ad013e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39474/0 (socket says 172.21.15.45:39474) 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad48b5640 1 -- 172.21.15.45:0/420536977 learned_addr learned my addr 172.21.15.45:0/420536977 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3452882027 0 0) 0x7f3ad009f7e0 con 0x7f3ad013c010 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient(hunting): my global_id is 134217 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ab0003530 con 0x7f3ad013c010 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 163514001 0 0) 0x7f3ad009b440 con 0x7f3ad0135950 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient(hunting): my global_id is 144116 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ad009f7e0 con 0x7f3ad0135950 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2314372223 0 0) 0x7f3ad009f7e0 con 0x7f3ad0135950 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 >> v1:172.21.15.46:6789/0 conn(0x7f3ad009fc40 legacy=0x7f3ad013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 >> v1:172.21.15.45:6789/0 conn(0x7f3ad013c010 legacy=0x7f3ad013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 monclient: found mon.noname-b 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad0136440 con 0x7f3ad0135950 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: _finish_auth 0 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3ab8001680 con 0x7f3ad0135950 2024-09-16T03:16:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: dump: 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.712 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 5 monclient: authenticate success, global_id 144116 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ab8001820 con 0x7f3ad0135950 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 1 -- 172.21.15.45:0/420536977 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3ab8001cb0 con 0x7f3ad0135950 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3acf7fe640 10 monclient: dump: 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 4 set_mon_vals no callback set 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 10 monclient: shutdown 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 -- 172.21.15.45:0/420536977 >> v1:172.21.15.45:6790/0 conn(0x7f3ad0135950 legacy=0x7f3ad0135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 -- 172.21.15.45:0/420536977 shutdown_connections 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 -- 172.21.15.45:0/420536977 >> 172.21.15.45:0/420536977 conn(0x7f3ad00a2bd0 msgr2=0x7f3ad00a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 -- 172.21.15.45:0/420536977 shutdown_connections 2024-09-16T03:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.698+0000 7f3ad6318640 1 -- 172.21.15.45:0/420536977 wait complete. 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient: build_initial_monmap 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient: monmap: 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 librados: starting msgr at 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 librados: starting objecter 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 Processor -- start 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 -- start start 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 librados: setting wanted keys 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 librados: calling monclient init 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient: init 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient(hunting): picked mon.noname-c con 0x7f3ad009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient(hunting): picked mon.noname-b con 0x7f3ad0135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient(hunting): picked mon.noname-a con 0x7f3ad013c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ad013b880 con 0x7f3ad013c010 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ad013f720 con 0x7f3ad0135950 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ad00714b0 con 0x7f3ad009fc40 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient(hunting): authenticate will time out at 1666.353435s 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3ad009fc40 0x7f3ad006e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37432/0 (socket says 172.21.15.45:37432) 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acffff640 1 -- 172.21.15.45:0/2578188085 learned_addr learned my addr 172.21.15.45:0/2578188085 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 client.?.objecter ms_handle_connect 0x7f3ad009fc40 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 client.?.objecter ms_handle_connect 0x7f3ad013c010 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 601891539 0 0) 0x7f3ad00714b0 con 0x7f3ad009fc40 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): my global_id is 144119 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ab4004ec0 con 0x7f3ad009fc40 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786615093 0 0) 0x7f3ac8001400 con 0x7f3ad013c010 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): my global_id is 134220 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ad00714b0 con 0x7f3ad013c010 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 client.?.objecter ms_handle_connect 0x7f3ad0135950 2024-09-16T03:16:15.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1159531629 0 0) 0x7f3ab4004ec0 con 0x7f3ad009fc40 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ab4006540 con 0x7f3ad009fc40 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3ab80036c0 con 0x7f3ad009fc40 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): dump: 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2576031264 0 0) 0x7f3ac0001090 con 0x7f3ad0135950 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): my global_id is 23122 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ab4004ec0 con 0x7f3ad0135950 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3926111694 0 0) 0x7f3ad00714b0 con 0x7f3ad013c010 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ac0001090 con 0x7f3ad013c010 2024-09-16T03:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3ac8001780 con 0x7f3ad013c010 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient(hunting): dump: 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3248585571 0 0) 0x7f3ab4006540 con 0x7f3ad009fc40 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 >> v1:172.21.15.46:6789/0 conn(0x7f3ad0135950 legacy=0x7f3ad006eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 >> v1:172.21.15.45:6789/0 conn(0x7f3ad013c010 legacy=0x7f3ad01d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient: _finish_hunting 0 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 monclient: found mon.c 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad006f1d0 con 0x7f3ad009fc40 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient: _finish_auth 0 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 5 monclient: authenticate success, global_id 144119 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient: _renew_subs 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad009d2c0 con 0x7f3ad009fc40 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 client.144119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient: _renew_subs 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ab8003b10 con 0x7f3ad009fc40 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ad013b010 con 0x7f3ad009fc40 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 librados: init done 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3ab8003ee0 con 0x7f3ad009fc40 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3aceffd640 4 set_mon_vals no callback set 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 monclient: dump: 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f3ab8004100 con 0x7f3ad009fc40 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(455..455 src has 99..455) ==== 5367+0+0 (unknown 3166234270 0 0) 0x7f3ab804b8a0 con 0x7f3ad009fc40 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 10 client.144119.objecter ms_dispatch 0x7f3ad0000ba0 osd_map(455..455 src has 99..455) 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 3 client.144119.objecter handle_osd_map got epochs [455,455] > 0 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 3 client.144119.objecter handle_osd_map decoding full epoch 455 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3aa80065d0 con 0x7f3aa8002870 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.702+0000 7f3acd7fa640 20 client.144119.objecter dump_active .. 0 homeless 2024-09-16T03:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3ac77fe640 10 monclient: get_auth_request con 0x7f3ab403e480 auth_method 0 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 client.144119.objecter ms_handle_connect 0x7f3aa8002870 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 client.144119.objecter resend_mon_ops 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 client.144119.objecter ms_handle_connect 0x7f3ab403e480 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1041799365 0 0) 0x7f3aa80065d0 con 0x7f3aa8002870 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 monclient(hunting): my global_id is 23125 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ab4006540 con 0x7f3aa8002870 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 568761143 0 0) 0x7f3ab4006540 con 0x7f3aa8002870 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3aa80065d0 con 0x7f3aa8002870 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3ac8004d10 con 0x7f3aa8002870 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 monclient: dump: 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4041139673 0 0) 0x7f3aa80065d0 con 0x7f3aa8002870 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3aa800a170 con 0x7f3aa8002870 2024-09-16T03:16:15.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3ac80062f0 con 0x7f3aa8002870 2024-09-16T03:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 >> v1:172.21.15.46:6789/0 conn(0x7f3aa8002870 legacy=0x7f3aa8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.706+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 >> v1:172.21.15.46:6789/0 conn(0x7f3aa8002870 legacy=0x7f3aa8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3aa8009e60 con 0x7f3aa8009120 2024-09-16T03:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 client.144119.objecter ms_handle_connect 0x7f3aa8009120 2024-09-16T03:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 client.144119.objecter resend_mon_ops 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216180718 0 0) 0x7f3aa8009e60 con 0x7f3aa8009120 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 monclient(hunting): my global_id is 23128 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3aa80065d0 con 0x7f3aa8009120 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3089518462 0 0) 0x7f3aa80065d0 con 0x7f3aa8009120 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3aa8009e60 con 0x7f3aa8009120 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3ac8004d10 con 0x7f3aa8009120 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 monclient: dump: 2024-09-16T03:16:15.730 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3992496391 0 0) 0x7f3aa8009e60 con 0x7f3aa8009120 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3aa800c3b0 con 0x7f3aa8009120 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 3785180958) 0x7f3ac80064a0 con 0x7f3aa8009120 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:16:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 >> v1:172.21.15.46:6789/0 conn(0x7f3aa8009120 legacy=0x7f3aa80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3acd7fa640 1 -- 172.21.15.45:0/2578188085 >> v1:172.21.15.46:6789/0 conn(0x7f3aa8009120 legacy=0x7f3aa80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 46, 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 697516, 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:16:15.732 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:15.733 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:15.734 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:15.735 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:15.736 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:15.737 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:15.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:16:15.739 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:16:15.740 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.741 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:15.742 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:15.743 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:15.744 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:15.744 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:15.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:15.744 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:16:15.744 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 10 librados: watch_flush enter 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 10 librados: watch_flush exit 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 20 client.144119.objecter shutdown clearing up homeless session... 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 10 client.144119.objecter successfully canceled tick 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 >> v1:172.21.15.46:6812/149360280 conn(0x7f3ab403e480 legacy=0x7f3ab4040940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 10 monclient: shutdown 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 >> v1:172.21.15.45:6790/0 conn(0x7f3ad009fc40 legacy=0x7f3ad006e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad48b5640 1 -- 172.21.15.45:0/2578188085 reap_dead start 2024-09-16T03:16:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad48b5640 1 -- 172.21.15.45:0/2578188085 reap_dead start 2024-09-16T03:16:15.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 shutdown_connections 2024-09-16T03:16:15.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 >> 172.21.15.45:0/2578188085 conn(0x7f3ad00a2bd0 msgr2=0x7f3ad013e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:15.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 shutdown_connections 2024-09-16T03:16:15.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 1 -- 172.21.15.45:0/2578188085 wait complete. 2024-09-16T03:16:15.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.714+0000 7f3ad6318640 1 librados: shutdown 2024-09-16T03:16:15.747 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:16:15.747 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:16:15.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:16:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.862+0000 7fb6d759e640 10 monclient: get_monmap_and_config 2024-09-16T03:16:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.862+0000 7fb6d759e640 10 monclient: build_initial_monmap 2024-09-16T03:16:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.862+0000 7fb6d759e640 10 monclient: monmap: 2024-09-16T03:16:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:15.867655+0000 2024-09-16T03:16:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:15.867655+0000 2024-09-16T03:16:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.867 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:15.867 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:15.867 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:15.867 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 Processor -- start 2024-09-16T03:16:15.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- start start 2024-09-16T03:16:15.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient: init 2024-09-16T03:16:15.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:15.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:16:15.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d80a0810 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:15.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient(hunting): picked mon.noname-b con 0x7fb6d806e0b0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:15.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient(hunting): picked mon.noname-a con 0x7fb6d80a1300 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:15.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d806eaa0 con 0x7fb6d80a1300 2024-09-16T03:16:15.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d80740b0 con 0x7fb6d80a0810 2024-09-16T03:16:15.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d806e7c0 con 0x7fb6d806e0b0 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient(hunting): authenticate will time out at 1666.517255s 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d6d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb6d80a1300 0x7fb6d813e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39490/0 (socket says 172.21.15.45:39490) 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d6d9d640 1 -- 172.21.15.45:0/2559143964 learned_addr learned my addr 172.21.15.45:0/2559143964 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 517959309 0 0) 0x7fb6c8001120 con 0x7fb6d80a1300 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient(hunting): my global_id is 134223 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6c4003530 con 0x7fb6d80a1300 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2495471513 0 0) 0x7fb6d806e7c0 con 0x7fb6d806e0b0 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient(hunting): my global_id is 144122 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6c8001120 con 0x7fb6d806e0b0 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3243372902 0 0) 0x7fb6c4003530 con 0x7fb6d80a1300 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 >> v1:172.21.15.45:6790/0 conn(0x7fb6d806e0b0 legacy=0x7fb6d813aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 >> v1:172.21.15.46:6789/0 conn(0x7fb6d80a0810 legacy=0x7fb6d80a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: _finish_hunting 0 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 monclient: found mon.noname-a 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:15.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d806ed80 con 0x7fb6d80a1300 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: _finish_auth 0 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: _check_auth_tickets 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb6c8001870 con 0x7fb6d80a1300 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 5 monclient: authenticate success, global_id 134223 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: dump: 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.880 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6c8001a90 con 0x7fb6d80a1300 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 1 -- 172.21.15.45:0/2559143964 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb6c80048b0 con 0x7fb6d80a1300 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 4 set_mon_vals no callback set 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d559a640 10 monclient: dump: 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.882 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient: shutdown 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- 172.21.15.45:0/2559143964 >> v1:172.21.15.45:6789/0 conn(0x7fb6d80a1300 legacy=0x7fb6d813e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- 172.21.15.45:0/2559143964 shutdown_connections 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- 172.21.15.45:0/2559143964 >> 172.21.15.45:0/2559143964 conn(0x7fb6d80a5fe0 msgr2=0x7fb6d80a63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- 172.21.15.45:0/2559143964 shutdown_connections 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- 172.21.15.45:0/2559143964 wait complete. 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient: build_initial_monmap 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient: monmap: 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:15.883 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 librados: starting msgr at 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 librados: starting objecter 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 Processor -- start 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 -- start start 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 librados: setting wanted keys 2024-09-16T03:16:15.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 1 librados: calling monclient init 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.866+0000 7fb6d759e640 10 monclient: init 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d806e0b0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient(hunting): picked mon.noname-b con 0x7fb6d80a0810 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient(hunting): picked mon.noname-a con 0x7fb6d80a1300 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d8124a70 con 0x7fb6d80a1300 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d8121a70 con 0x7fb6d80a0810 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d809ece0 con 0x7fb6d806e0b0 2024-09-16T03:16:15.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:15.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient(hunting): authenticate will time out at 1666.520313s 2024-09-16T03:16:15.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb6d806e0b0 0x7fb6d8120c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37450/0 (socket says 172.21.15.45:37450) 2024-09-16T03:16:15.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d659c640 1 -- 172.21.15.45:0/2517832804 learned_addr learned my addr 172.21.15.45:0/2517832804 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:15.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 client.?.objecter ms_handle_connect 0x7fb6d80a1300 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 client.?.objecter ms_handle_connect 0x7fb6d806e0b0 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 444700445 0 0) 0x7fb6d8124a70 con 0x7fb6d80a1300 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): my global_id is 134226 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6b8004ec0 con 0x7fb6d80a1300 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 283388549 0 0) 0x7fb6d809ece0 con 0x7fb6d806e0b0 2024-09-16T03:16:15.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): _negotiate 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): my global_id is 144125 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6d8124a70 con 0x7fb6d806e0b0 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 client.?.objecter ms_handle_connect 0x7fb6d80a0810 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4065335200 0 0) 0x7fb6b8004ec0 con 0x7fb6d80a1300 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6d809ece0 con 0x7fb6d80a1300 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb6c80047f0 con 0x7fb6d80a1300 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): dump: 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.889 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1587195894 0 0) 0x7fb6d8124a70 con 0x7fb6d806e0b0 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6b8004ec0 con 0x7fb6d806e0b0 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb6d00019a0 con 0x7fb6d806e0b0 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient(hunting): dump: 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.890 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.891 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.891 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.891 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.891 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.891 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 261045204 0 0) 0x7fb6d809ece0 con 0x7fb6d80a1300 2024-09-16T03:16:15.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 >> v1:172.21.15.45:6790/0 conn(0x7fb6d806e0b0 legacy=0x7fb6d8120c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 >> v1:172.21.15.46:6789/0 conn(0x7fb6d80a0810 legacy=0x7fb6d8121360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient: _finish_hunting 0 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 monclient: found mon.a 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d809d070 con 0x7fb6d80a1300 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient: _finish_auth 0 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient: _check_auth_tickets 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 5 monclient: authenticate success, global_id 134226 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient: _renew_subs 2024-09-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d8123c00 con 0x7fb6d80a1300 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 client.134226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient: _renew_subs 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6d81363d0 con 0x7fb6d80a1300 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 1 librados: init done 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6c8004e60 con 0x7fb6d80a1300 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb6c80012f0 con 0x7fb6d80a1300 2024-09-16T03:16:15.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 monclient: dump: 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:15.894 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6cffff640 4 set_mon_vals no callback set 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fb6c8010ee0 con 0x7fb6d80a1300 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(455..455 src has 99..455) ==== 5367+0+0 (unknown 3166234270 0 0) 0x7fb6c804b8b0 con 0x7fb6d80a1300 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 client.134226.objecter ms_dispatch 0x7fb6d8000ba0 osd_map(455..455 src has 99..455) 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 3 client.134226.objecter handle_osd_map got epochs [455,455] > 0 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 3 client.134226.objecter handle_osd_map decoding full epoch 455 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 20 client.134226.objecter dump_active .. 0 homeless 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d8124360 con 0x7fb6d80a1300 2024-09-16T03:16:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6d659c640 10 monclient: get_auth_request con 0x7fb6b803e030 auth_method 0 2024-09-16T03:16:15.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.870+0000 7fb6ceffd640 10 client.134226.objecter ms_handle_connect 0x7fb6b803e030 2024-09-16T03:16:15.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.874+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fb6d806eaa0 con 0x7fb6d80a1300 2024-09-16T03:16:15.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.874+0000 7fb6ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:15.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:15.874+0000 7fb6ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.114+0000 7fb6d759e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:16:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.114+0000 7fb6d759e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:16:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.114+0000 7fb6d759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.114+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb6d811db70 con 0x7fb6d80a1300 2024-09-16T03:16:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.118+0000 7fb6ceffd640 1 -- 172.21.15.45:0/2517832804 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fb6c8016d40 con 0x7fb6d80a1300 2024-09-16T03:16:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.118+0000 7fb6ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:16:16.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.118+0000 7fb6ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:16.120 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.118+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.120 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.118+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,health=12,kv=14,logm=54} crc {auth=2472151599,health=3932734053,kv=3374818887,logm=3867299351}) 2024-09-16T03:16:16.120 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.118+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=11,kv=13,logm=52} crc {auth=799813728,config=1097430493,health=4245724498,kv=3981266340,logm=1448732782}) 2024-09-16T03:16:16.120 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.118+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.120 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.118+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,health=12,kv=14,logm=54} crc {auth=2472151599,health=3932734053,kv=3374818887,logm=3867299351}) 2024-09-16T03:16:16.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.118+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=13,logm=53} crc {auth=2580584964,config=1097430493,health=3932734053,kv=268152317,logm=1336225081}) 2024-09-16T03:16:16.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210977191}) 2024-09-16T03:16:16.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2383062117}) 2024-09-16T03:16:16.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210977191}) 2024-09-16T03:16:16.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3847362335}) 2024-09-16T03:16:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 10 librados: watch_flush enter 2024-09-16T03:16:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 10 librados: watch_flush exit 2024-09-16T03:16:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 20 client.134226.objecter shutdown clearing up homeless session... 2024-09-16T03:16:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 10 client.134226.objecter successfully canceled tick 2024-09-16T03:16:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 >> v1:172.21.15.46:6812/149360280 conn(0x7fb6b803e030 legacy=0x7fb6b80404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 10 monclient: shutdown 2024-09-16T03:16:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 >> v1:172.21.15.45:6789/0 conn(0x7fb6d80a1300 legacy=0x7fb6d81d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 shutdown_connections 2024-09-16T03:16:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 >> 172.21.15.45:0/2517832804 conn(0x7fb6d80a5fe0 msgr2=0x7fb6d80a48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 shutdown_connections 2024-09-16T03:16:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 1 -- 172.21.15.45:0/2517832804 wait complete. 2024-09-16T03:16:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:16.122+0000 7fb6d759e640 1 librados: shutdown 2024-09-16T03:16:16.127 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.127 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818879207}) 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2807545276}) 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818879207}) 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.122+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3842224779}) 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1010947882}) 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4054663504}) 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1010947882}) 2024-09-16T03:16:16.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104951549}) 2024-09-16T03:16:16.131 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.131 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21880503}) 2024-09-16T03:16:16.131 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1752336018}) 2024-09-16T03:16:16.131 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.131 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21880503}) 2024-09-16T03:16:16.131 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.126+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991829545}) 2024-09-16T03:16:16.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819431522}) 2024-09-16T03:16:16.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389207981}) 2024-09-16T03:16:16.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819431522}) 2024-09-16T03:16:16.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=464388561}) 2024-09-16T03:16:16.133 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.133 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137785740}) 2024-09-16T03:16:16.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1386446167}) 2024-09-16T03:16:16.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137785740}) 2024-09-16T03:16:16.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.130+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3982215862}) 2024-09-16T03:16:16.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082102582}) 2024-09-16T03:16:16.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173521623}) 2024-09-16T03:16:16.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082102582}) 2024-09-16T03:16:16.137 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1220261295}) 2024-09-16T03:16:16.137 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.137 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84718976}) 2024-09-16T03:16:16.137 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2417233776}) 2024-09-16T03:16:16.138 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.138 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84718976}) 2024-09-16T03:16:16.138 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.134+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2097604971}) 2024-09-16T03:16:16.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648388218}) 2024-09-16T03:16:16.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2817126160}) 2024-09-16T03:16:16.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648388218}) 2024-09-16T03:16:16.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788296785}) 2024-09-16T03:16:16.140 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151052564}) 2024-09-16T03:16:16.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2586986586}) 2024-09-16T03:16:16.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151052564}) 2024-09-16T03:16:16.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.138+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021778061}) 2024-09-16T03:16:16.142 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3982517870}) 2024-09-16T03:16:16.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056516066}) 2024-09-16T03:16:16.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3982517870}) 2024-09-16T03:16:16.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4254070061}) 2024-09-16T03:16:16.144 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.144 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961685724}) 2024-09-16T03:16:16.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1070882000}) 2024-09-16T03:16:16.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961685724}) 2024-09-16T03:16:16.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=129527445}) 2024-09-16T03:16:16.146 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:16:16.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761565275}) 2024-09-16T03:16:16.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3480176814}) 2024-09-16T03:16:16.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761565275}) 2024-09-16T03:16:16.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.142+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228872016}) 2024-09-16T03:16:16.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436829619}) 2024-09-16T03:16:16.153 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1539802070}) 2024-09-16T03:16:16.153 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.153 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436829619}) 2024-09-16T03:16:16.153 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762991161}) 2024-09-16T03:16:16.155 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.155 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202834623}) 2024-09-16T03:16:16.155 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3467103751}) 2024-09-16T03:16:16.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202834623}) 2024-09-16T03:16:16.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.146+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2641653236}) 2024-09-16T03:16:16.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3190692322}) 2024-09-16T03:16:16.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996816716}) 2024-09-16T03:16:16.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3190692322}) 2024-09-16T03:16:16.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3102905008}) 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365358674}) 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4209852879}) 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365358674}) 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.150+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1283469846}) 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84756597}) 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107557805}) 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84756597}) 2024-09-16T03:16:16.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3700748086}) 2024-09-16T03:16:16.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997825935}) 2024-09-16T03:16:16.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874238331}) 2024-09-16T03:16:16.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997825935}) 2024-09-16T03:16:16.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.154+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1917399411}) 2024-09-16T03:16:16.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1024987676}) 2024-09-16T03:16:16.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1835262193}) 2024-09-16T03:16:16.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1024987676}) 2024-09-16T03:16:16.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308412750}) 2024-09-16T03:16:16.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574320657}) 2024-09-16T03:16:16.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=871947452}) 2024-09-16T03:16:16.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574320657}) 2024-09-16T03:16:16.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929398528}) 2024-09-16T03:16:16.163 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.163 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709128810}) 2024-09-16T03:16:16.163 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863504459}) 2024-09-16T03:16:16.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709128810}) 2024-09-16T03:16:16.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.158+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845283156}) 2024-09-16T03:16:16.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794651418}) 2024-09-16T03:16:16.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2746232879}) 2024-09-16T03:16:16.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794651418}) 2024-09-16T03:16:16.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160941637}) 2024-09-16T03:16:16.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284161596}) 2024-09-16T03:16:16.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2806073032}) 2024-09-16T03:16:16.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284161596}) 2024-09-16T03:16:16.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.162+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3738948585}) 2024-09-16T03:16:16.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800725756}) 2024-09-16T03:16:16.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553082699}) 2024-09-16T03:16:16.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800725756}) 2024-09-16T03:16:16.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3269962790}) 2024-09-16T03:16:16.168 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.168 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=752662965}) 2024-09-16T03:16:16.168 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3457265591}) 2024-09-16T03:16:16.168 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.169 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=752662965}) 2024-09-16T03:16:16.169 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3171524700}) 2024-09-16T03:16:16.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135011036}) 2024-09-16T03:16:16.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3425579718}) 2024-09-16T03:16:16.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135011036}) 2024-09-16T03:16:16.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.166+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316851346}) 2024-09-16T03:16:16.171 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473639551}) 2024-09-16T03:16:16.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=242359710}) 2024-09-16T03:16:16.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473639551}) 2024-09-16T03:16:16.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4036944650}) 2024-09-16T03:16:16.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661658603}) 2024-09-16T03:16:16.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=611107405}) 2024-09-16T03:16:16.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661658603}) 2024-09-16T03:16:16.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.170+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2251691410}) 2024-09-16T03:16:16.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676573851}) 2024-09-16T03:16:16.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3620918045}) 2024-09-16T03:16:16.176 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.176 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676573851}) 2024-09-16T03:16:16.176 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3314538776}) 2024-09-16T03:16:16.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710820927}) 2024-09-16T03:16:16.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3900216589}) 2024-09-16T03:16:16.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710820927}) 2024-09-16T03:16:16.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.174+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=896686395}) 2024-09-16T03:16:16.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41298151}) 2024-09-16T03:16:16.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911093439}) 2024-09-16T03:16:16.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.180 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41298151}) 2024-09-16T03:16:16.180 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134169391}) 2024-09-16T03:16:16.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302161896}) 2024-09-16T03:16:16.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947296181}) 2024-09-16T03:16:16.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302161896}) 2024-09-16T03:16:16.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1387809966}) 2024-09-16T03:16:16.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=671916266}) 2024-09-16T03:16:16.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3090888821}) 2024-09-16T03:16:16.183 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.183 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=671916266}) 2024-09-16T03:16:16.183 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.178+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1377842038}) 2024-09-16T03:16:16.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889276788}) 2024-09-16T03:16:16.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2231260266}) 2024-09-16T03:16:16.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889276788}) 2024-09-16T03:16:16.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235849249}) 2024-09-16T03:16:16.185 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802173805}) 2024-09-16T03:16:16.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1409339488}) 2024-09-16T03:16:16.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802173805}) 2024-09-16T03:16:16.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.182+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3631467324}) 2024-09-16T03:16:16.187 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.187 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280993104}) 2024-09-16T03:16:16.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1567325701}) 2024-09-16T03:16:16.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280993104}) 2024-09-16T03:16:16.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2246927550}) 2024-09-16T03:16:16.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422208397}) 2024-09-16T03:16:16.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1472106061}) 2024-09-16T03:16:16.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422208397}) 2024-09-16T03:16:16.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.186+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1826062559}) 2024-09-16T03:16:16.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778305682}) 2024-09-16T03:16:16.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2566397225}) 2024-09-16T03:16:16.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778305682}) 2024-09-16T03:16:16.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=511661980}) 2024-09-16T03:16:16.192 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2866947111}) 2024-09-16T03:16:16.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4201346136}) 2024-09-16T03:16:16.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2866947111}) 2024-09-16T03:16:16.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.190+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1302297714}) 2024-09-16T03:16:16.194 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.194 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381184481}) 2024-09-16T03:16:16.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=316297668}) 2024-09-16T03:16:16.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381184481}) 2024-09-16T03:16:16.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=173209038}) 2024-09-16T03:16:16.196 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.196 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551214335}) 2024-09-16T03:16:16.196 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568351034}) 2024-09-16T03:16:16.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551214335}) 2024-09-16T03:16:16.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3599577462}) 2024-09-16T03:16:16.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3913716883}) 2024-09-16T03:16:16.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345006803}) 2024-09-16T03:16:16.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3913716883}) 2024-09-16T03:16:16.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.194+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=134531307}) 2024-09-16T03:16:16.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519135644}) 2024-09-16T03:16:16.200 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=738701174}) 2024-09-16T03:16:16.200 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.200 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519135644}) 2024-09-16T03:16:16.200 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2302531627}) 2024-09-16T03:16:16.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110241396}) 2024-09-16T03:16:16.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=591736559}) 2024-09-16T03:16:16.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110241396}) 2024-09-16T03:16:16.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.198+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1126564308}) 2024-09-16T03:16:16.202 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841003430}) 2024-09-16T03:16:16.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220839442}) 2024-09-16T03:16:16.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841003430}) 2024-09-16T03:16:16.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=889510011}) 2024-09-16T03:16:16.204 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.204 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=234394672}) 2024-09-16T03:16:16.205 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694572732}) 2024-09-16T03:16:16.205 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.205 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=234394672}) 2024-09-16T03:16:16.205 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1384951972}) 2024-09-16T03:16:16.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729753325}) 2024-09-16T03:16:16.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1325082648}) 2024-09-16T03:16:16.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729753325}) 2024-09-16T03:16:16.207 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.202+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=221495869}) 2024-09-16T03:16:16.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.206+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.206+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=13,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=31,osdmap=4} crc {logm=950275559,mdsmap=3140514009,mgr=3580333927,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1447737124,mon_config_key=3941689943,monmap=3372108098,nvmeofgw=598337686,osd_metadata=773870977,osd_pg_creating=2389567977,osd_snap=3617848917,osdmap=3330332510}) 2024-09-16T03:16:16.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.206+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=14,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31} crc {logm=1760591014,mdsmap=2287870078,mgr=715532488,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=703577867,mon_config_key=305327012,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=2389567977,osd_snap=2606772888}) 2024-09-16T03:16:16.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.206+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.206+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=13,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=31,osdmap=4} crc {logm=950275559,mdsmap=3140514009,mgr=3580333927,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1447737124,mon_config_key=3941689943,monmap=3372108098,nvmeofgw=598337686,osd_metadata=773870977,osd_pg_creating=2389567977,osd_snap=3617848917,osdmap=3330332510}) 2024-09-16T03:16:16.209 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.206+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=14,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=2,nvmeofgw=1,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=2} crc {logm=1760591014,mdsmap=2287870078,mgr=3580333927,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=2048708611,mon_config_key=2521979059,monmap=127340147,nvmeofgw=1228700967,osd_metadata=2004720307,osd_pg_creating=2389567977,osd_snap=1278617496,osdmap=1266944524}) 2024-09-16T03:16:16.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2726528979}) 2024-09-16T03:16:16.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1487695099}) 2024-09-16T03:16:16.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.211 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2726528979}) 2024-09-16T03:16:16.211 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3821558762}) 2024-09-16T03:16:16.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3713985118}) 2024-09-16T03:16:16.213 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4077139108}) 2024-09-16T03:16:16.213 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.213 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3713985118}) 2024-09-16T03:16:16.213 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1871863099}) 2024-09-16T03:16:16.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.210+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3426487347}) 2024-09-16T03:16:16.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3668961051}) 2024-09-16T03:16:16.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.215 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3426487347}) 2024-09-16T03:16:16.215 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=993575688}) 2024-09-16T03:16:16.217 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.217 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3662958126}) 2024-09-16T03:16:16.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1038233573}) 2024-09-16T03:16:16.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3662958126}) 2024-09-16T03:16:16.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.214+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=595859771}) 2024-09-16T03:16:16.222 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.218+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.222 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.218+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3283236285}) 2024-09-16T03:16:16.222 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.218+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3280098190}) 2024-09-16T03:16:16.222 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.218+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.222 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.218+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3283236285}) 2024-09-16T03:16:16.222 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.218+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=997636959}) 2024-09-16T03:16:16.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.230+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.230+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3753675672}) 2024-09-16T03:16:16.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.230+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=22} crc {osdmap=421458759}) 2024-09-16T03:16:16.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.230+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:16:16.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.230+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3753675672}) 2024-09-16T03:16:16.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:16:16.230+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=22} crc {osdmap=2053549532}) 2024-09-16T03:16:17.147 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:16:17.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:16:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient: get_monmap_and_config 2024-09-16T03:16:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient: build_initial_monmap 2024-09-16T03:16:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient: monmap: 2024-09-16T03:16:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:17.281658+0000 2024-09-16T03:16:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:17.281658+0000 2024-09-16T03:16:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:17.281 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.281 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:17.281 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:17.281 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:17.281 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 1 Processor -- start 2024-09-16T03:16:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 1 -- start start 2024-09-16T03:16:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient: init 2024-09-16T03:16:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:16:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient(hunting): picked mon.noname-a con 0x7f063c09b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient(hunting): picked mon.noname-b con 0x7f063c09b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient(hunting): picked mon.noname-c con 0x7f063c13b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f063c09cc20 con 0x7f063c09b8e0 2024-09-16T03:16:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f063c09d820 con 0x7f063c13b000 2024-09-16T03:16:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f063c09b6a0 con 0x7f063c09b250 2024-09-16T03:16:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 10 monclient(hunting): authenticate will time out at 1667.931009s 2024-09-16T03:16:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641c79640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f063c09b250 0x7f063c1398e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37462/0 (socket says 172.21.15.45:37462) 2024-09-16T03:16:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641c79640 1 -- 172.21.15.45:0/1207172556 learned_addr learned my addr 172.21.15.45:0/1207172556 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f064247a640 1 --1- 172.21.15.45:0/1207172556 >> v1:172.21.15.45:6789/0 conn(0x7f063c09b8e0 0x7f063c09bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39504/0 (socket says 172.21.15.45:39504) 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1963240232 0 0) 0x7f063c09b6a0 con 0x7f063c09b250 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient(hunting): my global_id is 144128 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0628003530 con 0x7f063c09b250 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005003632 0 0) 0x7f063c09cc20 con 0x7f063c09b8e0 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient(hunting): my global_id is 134229 2024-09-16T03:16:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f063c09b6a0 con 0x7f063c09b8e0 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3189901136 0 0) 0x7f063c09b6a0 con 0x7f063c09b8e0 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 >> v1:172.21.15.45:6790/0 conn(0x7f063c09b250 legacy=0x7f063c1398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 >> v1:172.21.15.46:6789/0 conn(0x7f063c13b000 legacy=0x7f063c13d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient: _finish_hunting 0 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 monclient: found mon.noname-a 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063c0a21f0 con 0x7f063c09b8e0 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient: _finish_auth 0 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient: _check_auth_tickets 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0638001a00 con 0x7f063c09b8e0 2024-09-16T03:16:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 5 monclient: authenticate success, global_id 134229 2024-09-16T03:16:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f0641478640 10 monclient: dump: 2024-09-16T03:16:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.278+0000 7f06446de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0638001ba0 con 0x7f063c09b8e0 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0641478640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0641478640 1 -- 172.21.15.45:0/1207172556 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0638001fb0 con 0x7f063c09b8e0 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 4 set_mon_vals no callback set 2024-09-16T03:16:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0641478640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0641478640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0641478640 10 monclient: dump: 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: shutdown 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- 172.21.15.45:0/1207172556 >> v1:172.21.15.45:6789/0 conn(0x7f063c09b8e0 legacy=0x7f063c09bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- 172.21.15.45:0/1207172556 shutdown_connections 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- 172.21.15.45:0/1207172556 >> 172.21.15.45:0/1207172556 conn(0x7f063c0a2bd0 msgr2=0x7f063c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- 172.21.15.45:0/1207172556 shutdown_connections 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- 172.21.15.45:0/1207172556 wait complete. 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: build_initial_monmap 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: monmap: 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 librados: starting msgr at 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 librados: starting objecter 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 Processor -- start 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- start start 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 librados: setting wanted keys 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 librados: calling monclient init 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: init 2024-09-16T03:16:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient(hunting): picked mon.noname-c con 0x7f063c09b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient(hunting): picked mon.noname-b con 0x7f063c09b8e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient(hunting): picked mon.noname-a con 0x7f063c13b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f063c09ae10 con 0x7f063c13b000 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f063c13ccd0 con 0x7f063c09b8e0 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f063c094ca0 con 0x7f063c09b250 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient(hunting): authenticate will time out at 1667.934075s 2024-09-16T03:16:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f064247a640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f063c09b250 0x7f063c1ccf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37466/0 (socket says 172.21.15.45:37466) 2024-09-16T03:16:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0642c7b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f063c13b000 0x7f063c1d56c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39518/0 (socket says 172.21.15.45:39518) 2024-09-16T03:16:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f064247a640 1 -- 172.21.15.45:0/3838201735 learned_addr learned my addr 172.21.15.45:0/3838201735 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 client.?.objecter ms_handle_connect 0x7f063c13b000 2024-09-16T03:16:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 client.?.objecter ms_handle_connect 0x7f063c09b250 2024-09-16T03:16:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 155105504 0 0) 0x7f063c094ca0 con 0x7f063c09b250 2024-09-16T03:16:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): my global_id is 144131 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0620004ec0 con 0x7f063c09b250 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1271017538 0 0) 0x7f063c09ae10 con 0x7f063c13b000 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): my global_id is 134232 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f063c094ca0 con 0x7f063c13b000 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 client.?.objecter ms_handle_connect 0x7f063c09b8e0 2024-09-16T03:16:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3258789868 0 0) 0x7f063c094ca0 con 0x7f063c13b000 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f063c09ae10 con 0x7f063c13b000 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1523395075 0 0) 0x7f0620004ec0 con 0x7f063c09b250 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f063c094ca0 con 0x7f063c09b250 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0634001840 con 0x7f063c13b000 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): dump: 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0638001f70 con 0x7f063c09b250 2024-09-16T03:16:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): dump: 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.303 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.303 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3587719541 0 0) 0x7f063c13ccd0 con 0x7f063c09b8e0 2024-09-16T03:16:17.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient(hunting): my global_id is 23140 2024-09-16T03:16:17.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0620004ec0 con 0x7f063c09b8e0 2024-09-16T03:16:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 658856725 0 0) 0x7f063c094ca0 con 0x7f063c09b250 2024-09-16T03:16:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 >> v1:172.21.15.46:6789/0 conn(0x7f063c09b8e0 legacy=0x7f063c1d1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 >> v1:172.21.15.45:6789/0 conn(0x7f063c13b000 legacy=0x7f063c1d56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:16:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 monclient: found mon.c 2024-09-16T03:16:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063c0934a0 con 0x7f063c09b250 2024-09-16T03:16:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient: _finish_auth 0 2024-09-16T03:16:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:16:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 5 monclient: authenticate success, global_id 144131 2024-09-16T03:16:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: _renew_subs 2024-09-16T03:16:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f063c0956a0 con 0x7f063c09b250 2024-09-16T03:16:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 client.144131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: _renew_subs 2024-09-16T03:16:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f063c13c5c0 con 0x7f063c09b250 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f06446de640 1 librados: init done 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0638004e20 con 0x7f063c09b250 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f06380052b0 con 0x7f063c09b250 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0640c77640 4 set_mon_vals no callback set 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.282+0000 7f0632ffd640 10 monclient: dump: 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.307 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.307 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.307 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.307 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.307 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.307 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.307 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.307 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f06446de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f06446de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f06446de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0624005090 con 0x7f063c09b250 2024-09-16T03:16:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f06380054d0 con 0x7f063c09b250 2024-09-16T03:16:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(457..457 src has 99..457) ==== 5367+0+0 (unknown 3307601601 0 0) 0x7f063804bb10 con 0x7f063c09b250 2024-09-16T03:16:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f0632ffd640 10 client.144131.objecter ms_dispatch 0x7f063c000ba0 osd_map(457..457 src has 99..457) 2024-09-16T03:16:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f0632ffd640 3 client.144131.objecter handle_osd_map got epochs [457,457] > 0 2024-09-16T03:16:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f0632ffd640 3 client.144131.objecter handle_osd_map decoding full epoch 457 2024-09-16T03:16:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f0632ffd640 20 client.144131.objecter dump_active .. 0 homeless 2024-09-16T03:16:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f0641c79640 10 monclient: get_auth_request con 0x7f062003e1d0 auth_method 0 2024-09-16T03:16:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.286+0000 7f0632ffd640 10 client.144131.objecter ms_handle_connect 0x7f062003e1d0 2024-09-16T03:16:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.290+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f0638016ec0 con 0x7f063c09b250 2024-09-16T03:16:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.290+0000 7f0632ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.290+0000 7f0632ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.530+0000 7f06446de640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:16:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.530+0000 7f06446de640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.530+0000 7f06446de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.530+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0624005380 con 0x7f063c09b250 2024-09-16T03:16:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.530+0000 7f0632ffd640 1 -- 172.21.15.45:0/3838201735 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 1142564077) 0x7f06380167c0 con 0x7f063c09b250 2024-09-16T03:16:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.530+0000 7f0632ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.530+0000 7f0632ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:17.533 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 10 librados: watch_flush enter 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 10 librados: watch_flush exit 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 20 client.144131.objecter shutdown clearing up homeless session... 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 10 client.144131.objecter successfully canceled tick 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 >> v1:172.21.15.46:6812/149360280 conn(0x7f062003e1d0 legacy=0x7f0620040690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 10 monclient: shutdown 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 >> v1:172.21.15.45:6790/0 conn(0x7f063c09b250 legacy=0x7f063c1ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 shutdown_connections 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 >> 172.21.15.45:0/3838201735 conn(0x7f063c0a2bd0 msgr2=0x7f063c0a5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 shutdown_connections 2024-09-16T03:16:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 1 -- 172.21.15.45:0/3838201735 wait complete. 2024-09-16T03:16:17.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.534+0000 7f06446de640 1 librados: shutdown 2024-09-16T03:16:17.551 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:16:17.551 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:16:17.551 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:16:17.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:16:17.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.662+0000 7f61bb862640 10 monclient: get_monmap_and_config 2024-09-16T03:16:17.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.662+0000 7f61bb862640 10 monclient: build_initial_monmap 2024-09-16T03:16:17.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.662+0000 7f61bb862640 10 monclient: monmap: 2024-09-16T03:16:17.666 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:17.666 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:17.668066+0000 2024-09-16T03:16:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:17.668066+0000 2024-09-16T03:16:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:17.667 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 Processor -- start 2024-09-16T03:16:17.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 -- start start 2024-09-16T03:16:17.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient: init 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient(hunting): picked mon.noname-a con 0x7f61b40a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient(hunting): picked mon.noname-c con 0x7f61b40a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient(hunting): picked mon.noname-b con 0x7f61b4073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b409d9e0 con 0x7f61b40a4d20 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b409dc20 con 0x7f61b40a5810 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b4073360 con 0x7f61b4073d40 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient(hunting): authenticate will time out at 1668.317225s 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61b95fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f61b40a4d20 0x7f61b40a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39532/0 (socket says 172.21.15.45:39532) 2024-09-16T03:16:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61b95fe640 1 -- 172.21.15.45:0/2819730722 learned_addr learned my addr 172.21.15.45:0/2819730722 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1154359049 0 0) 0x7f61b409d9e0 con 0x7f61b40a4d20 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient(hunting): my global_id is 134235 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61a0003530 con 0x7f61b40a4d20 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3708389200 0 0) 0x7f61b4073360 con 0x7f61b4073d40 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient(hunting): my global_id is 144134 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61b409d9e0 con 0x7f61b4073d40 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1854505921 0 0) 0x7f61a0003530 con 0x7f61b40a4d20 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 >> v1:172.21.15.45:6790/0 conn(0x7f61b4073d40 legacy=0x7f61b4140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 >> v1:172.21.15.46:6789/0 conn(0x7f61b40a5810 legacy=0x7f61b413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: _finish_hunting 0 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 monclient: found mon.noname-a 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b409cf30 con 0x7f61b40a4d20 2024-09-16T03:16:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: _finish_auth 0 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: _check_auth_tickets 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f61b00016a0 con 0x7f61b40a4d20 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: dump: 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.679 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 5 monclient: authenticate success, global_id 134235 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61b0001840 con 0x7f61b40a4d20 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 1 -- 172.21.15.45:0/2819730722 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f61b0001c90 con 0x7f61b40a4d20 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61abfff640 10 monclient: dump: 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 4 set_mon_vals no callback set 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient: shutdown 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 -- 172.21.15.45:0/2819730722 >> v1:172.21.15.45:6789/0 conn(0x7f61b40a4d20 legacy=0x7f61b40a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 -- 172.21.15.45:0/2819730722 shutdown_connections 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 -- 172.21.15.45:0/2819730722 >> 172.21.15.45:0/2819730722 conn(0x7f61b40aa720 msgr2=0x7f61b40aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 -- 172.21.15.45:0/2819730722 shutdown_connections 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 -- 172.21.15.45:0/2819730722 wait complete. 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient: build_initial_monmap 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 10 monclient: monmap: 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 librados: starting msgr at 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 librados: starting objecter 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.666+0000 7f61bb862640 1 Processor -- start 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 -- start start 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 librados: setting wanted keys 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 librados: calling monclient init 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient: init 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient(hunting): picked mon.noname-b con 0x7f61b4073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient(hunting): picked mon.noname-a con 0x7f61b40a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:17.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient(hunting): picked mon.noname-c con 0x7f61b40a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b4140650 con 0x7f61b40a4d20 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b4140930 con 0x7f61b4073d40 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b40a1510 con 0x7f61b40a5810 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient(hunting): authenticate will time out at 1668.320625s 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61b8dfd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f61b40a4d20 0x7f61b41d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39548/0 (socket says 172.21.15.45:39548) 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61b8dfd640 1 -- 172.21.15.45:0/1407033634 learned_addr learned my addr 172.21.15.45:0/1407033634 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.?.objecter ms_handle_connect 0x7f61b40a5810 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.?.objecter ms_handle_connect 0x7f61b40a4d20 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675602280 0 0) 0x7f61b40a1510 con 0x7f61b40a5810 2024-09-16T03:16:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): my global_id is 144137 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6198004ec0 con 0x7f61b40a5810 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832810023 0 0) 0x7f61b4140650 con 0x7f61b40a4d20 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): my global_id is 134238 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61b40a1510 con 0x7f61b40a4d20 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.?.objecter ms_handle_connect 0x7f61b4073d40 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2556837348 0 0) 0x7f6198004ec0 con 0x7f61b40a5810 2024-09-16T03:16:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61b4140650 con 0x7f61b40a5810 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2073212396 0 0) 0x7f61b40a1510 con 0x7f61b40a4d20 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6198004ec0 con 0x7f61b40a4d20 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f61ac001cc0 con 0x7f61b40a5810 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): dump: 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f61a4001980 con 0x7f61b40a4d20 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): dump: 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3125527508 0 0) 0x7f61b4140930 con 0x7f61b4073d40 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): my global_id is 23146 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61b40a1510 con 0x7f61b4073d40 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1155049860 0 0) 0x7f61b4140650 con 0x7f61b40a5810 2024-09-16T03:16:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 >> v1:172.21.15.46:6789/0 conn(0x7f61b4073d40 legacy=0x7f61b41377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 >> v1:172.21.15.45:6789/0 conn(0x7f61b40a4d20 legacy=0x7f61b41d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 monclient: found mon.c 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b4140370 con 0x7f61b40a5810 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 5 monclient: authenticate success, global_id 144137 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient: _renew_subs 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f61b4137ee0 con 0x7f61b40a5810 2024-09-16T03:16:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 client.144137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient: _renew_subs 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f61b413dcf0 con 0x7f61b40a5810 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 librados: init done 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61ac003980 con 0x7f61b40a5810 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f61ac003cd0 con 0x7f61b40a5810 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61ab7fe640 4 set_mon_vals no callback set 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: dump: 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61940065d0 con 0x7f6194002870 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.144137.objecter ms_handle_connect 0x7f6194002870 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.144137.objecter resend_mon_ops 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f61ac00fd40 con 0x7f61b40a5810 2024-09-16T03:16:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(457..457 src has 99..457) ==== 5367+0+0 (unknown 3307601601 0 0) 0x7f61ac04a840 con 0x7f61b40a5810 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.144137.objecter ms_dispatch 0x7f61b4000ba0 osd_map(457..457 src has 99..457) 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 3 client.144137.objecter handle_osd_map got epochs [457,457] > 0 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 3 client.144137.objecter handle_osd_map decoding full epoch 457 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 20 client.144137.objecter dump_active .. 0 homeless 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3108169029 0 0) 0x7f61940065d0 con 0x7f6194002870 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient(hunting): my global_id is 134241 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61b4140650 con 0x7f6194002870 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3750160732 0 0) 0x7f61b4140650 con 0x7f6194002870 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61940065d0 con 0x7f6194002870 2024-09-16T03:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f61b00015e0 con 0x7f6194002870 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: dump: 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61b8dfd640 10 monclient: get_auth_request con 0x7f619803e1e0 auth_method 0 2024-09-16T03:16:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 394203083 0 0) 0x7f61940065d0 con 0x7f6194002870 2024-09-16T03:16:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f619400a170 con 0x7f6194002870 2024-09-16T03:16:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 client.144137.objecter ms_handle_connect 0x7f619803e1e0 2024-09-16T03:16:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f61b00018c0 con 0x7f6194002870 2024-09-16T03:16:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 >> v1:172.21.15.45:6789/0 conn(0x7f6194002870 legacy=0x7f6194004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.670+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 >> v1:172.21.15.45:6789/0 conn(0x7f6194002870 legacy=0x7f6194004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6194009e80 con 0x7f6194009140 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 client.144137.objecter ms_handle_connect 0x7f6194009140 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 client.144137.objecter resend_mon_ops 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085522635 0 0) 0x7f6194009e80 con 0x7f6194009140 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 monclient(hunting): my global_id is 134244 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61940065d0 con 0x7f6194009140 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3882226725 0 0) 0x7f61940065d0 con 0x7f6194009140 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6194009e80 con 0x7f6194009140 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f61b0004af0 con 0x7f6194009140 2024-09-16T03:16:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 monclient: dump: 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3944944795 0 0) 0x7f6194009e80 con 0x7f6194009140 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f619400c420 con 0x7f6194009140 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4137 (unknown 0 0 3160435397) 0x7f61b0004950 con 0x7f6194009140 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 >> v1:172.21.15.45:6789/0 conn(0x7f6194009140 legacy=0x7f61940095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61a9ffb640 1 -- 172.21.15.45:0/1407033634 >> v1:172.21.15.45:6789/0 conn(0x7f6194009140 legacy=0x7f61940095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:17.698 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 46, 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 110906, 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:16:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:16:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.710 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:16:17.710 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:16:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 10 librados: watch_flush enter 2024-09-16T03:16:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 10 librados: watch_flush exit 2024-09-16T03:16:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 20 client.144137.objecter shutdown clearing up homeless session... 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 10 client.144137.objecter successfully canceled tick 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 >> v1:172.21.15.46:6812/149360280 conn(0x7f619803e1e0 legacy=0x7f61980406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 10 monclient: shutdown 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 >> v1:172.21.15.45:6790/0 conn(0x7f61b40a5810 legacy=0x7f61b41d58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61b9dff640 1 -- 172.21.15.45:0/1407033634 reap_dead start 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61b9dff640 1 -- 172.21.15.45:0/1407033634 reap_dead start 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 shutdown_connections 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 >> 172.21.15.45:0/1407033634 conn(0x7f61b40aa720 msgr2=0x7f61b40a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 shutdown_connections 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 1 -- 172.21.15.45:0/1407033634 wait complete. 2024-09-16T03:16:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.678+0000 7f61bb862640 1 librados: shutdown 2024-09-16T03:16:17.713 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:16:17.713 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:16:17.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:16:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient: get_monmap_and_config 2024-09-16T03:16:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient: build_initial_monmap 2024-09-16T03:16:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient: monmap: 2024-09-16T03:16:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:17.824892+0000 2024-09-16T03:16:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:17.824892+0000 2024-09-16T03:16:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.824 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:17.824 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:17.824 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:17.824 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 Processor -- start 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 -- start start 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient: init 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef0135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef013c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef009f7e0 con 0x7f5ef009fc40 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef009b160 con 0x7f5ef0135950 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef009b440 con 0x7f5ef013c010 2024-09-16T03:16:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient(hunting): authenticate will time out at 1668.474236s 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef658c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5ef013c010 0x7f5ef013e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37492/0 (socket says 172.21.15.45:37492) 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef658c640 1 -- 172.21.15.45:0/381618943 learned_addr learned my addr 172.21.15.45:0/381618943 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef558a640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5ef009fc40 0x7f5ef013a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39570/0 (socket says 172.21.15.45:39570) 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1471031504 0 0) 0x7f5ef009b440 con 0x7f5ef013c010 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient(hunting): my global_id is 144140 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed8003530 con 0x7f5ef013c010 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250494473 0 0) 0x7f5ef009f7e0 con 0x7f5ef009fc40 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient(hunting): my global_id is 134247 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef009b440 con 0x7f5ef009fc40 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2775317532 0 0) 0x7f5ed8003530 con 0x7f5ef013c010 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 >> v1:172.21.15.46:6789/0 conn(0x7f5ef0135950 legacy=0x7f5ef0135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 >> v1:172.21.15.45:6789/0 conn(0x7f5ef009fc40 legacy=0x7f5ef013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: _finish_hunting 0 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 monclient: found mon.noname-b 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef0136440 con 0x7f5ef013c010 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: _finish_auth 0 2024-09-16T03:16:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: _check_auth_tickets 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5ee00016a0 con 0x7f5ef013c010 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 5 monclient: authenticate success, global_id 144140 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: dump: 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ee0001840 con 0x7f5ef013c010 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 1 -- 172.21.15.45:0/381618943 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5ee0001cd0 con 0x7f5ef013c010 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef4d89640 10 monclient: dump: 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 4 set_mon_vals no callback set 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 10 monclient: shutdown 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 -- 172.21.15.45:0/381618943 >> v1:172.21.15.45:6790/0 conn(0x7f5ef013c010 legacy=0x7f5ef013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 -- 172.21.15.45:0/381618943 shutdown_connections 2024-09-16T03:16:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 -- 172.21.15.45:0/381618943 >> 172.21.15.45:0/381618943 conn(0x7f5ef00a2bd0 msgr2=0x7f5ef00a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 -- 172.21.15.45:0/381618943 shutdown_connections 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.822+0000 7f5ef7fef640 1 -- 172.21.15.45:0/381618943 wait complete. 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient: build_initial_monmap 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient: monmap: 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:17.834 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 librados: starting msgr at 2024-09-16T03:16:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 librados: starting objecter 2024-09-16T03:16:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 Processor -- start 2024-09-16T03:16:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 -- start start 2024-09-16T03:16:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 librados: setting wanted keys 2024-09-16T03:16:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 librados: calling monclient init 2024-09-16T03:16:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient: init 2024-09-16T03:16:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:16:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef0135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef0070ed0 con 0x7f5ef009fc40 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef013bb60 con 0x7f5ef013c010 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef00711c0 con 0x7f5ef0135950 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient(hunting): authenticate will time out at 1668.477392s 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef558a640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5ef0135950 0x7f5ef006eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37500/0 (socket says 172.21.15.45:37500) 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef558a640 1 -- 172.21.15.45:0/1723932260 learned_addr learned my addr 172.21.15.45:0/1723932260 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 client.?.objecter ms_handle_connect 0x7f5ef009fc40 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 client.?.objecter ms_handle_connect 0x7f5ef0135950 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3356664776 0 0) 0x7f5ef00711c0 con 0x7f5ef0135950 2024-09-16T03:16:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): my global_id is 144143 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5edc004ec0 con 0x7f5ef0135950 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2117134728 0 0) 0x7f5ef0070ed0 con 0x7f5ef009fc40 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): my global_id is 134250 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef00711c0 con 0x7f5ef009fc40 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 client.?.objecter ms_handle_connect 0x7f5ef013c010 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1388103624 0 0) 0x7f5edc004ec0 con 0x7f5ef0135950 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ef0070ed0 con 0x7f5ef0135950 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5eec0019d0 con 0x7f5ef0135950 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): dump: 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 945918333 0 0) 0x7f5ef00711c0 con 0x7f5ef009fc40 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5edc004ec0 con 0x7f5ef009fc40 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5ee8001bc0 con 0x7f5ef009fc40 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): dump: 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3091078855 0 0) 0x7f5ef0070ed0 con 0x7f5ef0135950 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 >> v1:172.21.15.46:6789/0 conn(0x7f5ef013c010 legacy=0x7f5ef006f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 >> v1:172.21.15.45:6789/0 conn(0x7f5ef009fc40 legacy=0x7f5ef006e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: _finish_hunting 0 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 monclient: found mon.c 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef00714a0 con 0x7f5ef0135950 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: _finish_auth 0 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 5 monclient: authenticate success, global_id 144143 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient: _renew_subs 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef0070430 con 0x7f5ef0135950 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 client.144143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient: _renew_subs 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ef013b6f0 con 0x7f5ef0135950 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 librados: init done 2024-09-16T03:16:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5eec0047f0 con 0x7f5ef0135950 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5eec004b40 con 0x7f5ef0135950 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee7fff640 4 set_mon_vals no callback set 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: dump: 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ed00065d0 con 0x7f5ed0002870 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 client.144143.objecter ms_handle_connect 0x7f5ed0002870 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 client.144143.objecter resend_mon_ops 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f5eec010bb0 con 0x7f5ef0135950 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(457..457 src has 99..457) ==== 5367+0+0 (unknown 3307601601 0 0) 0x7f5eec04b7d0 con 0x7f5ef0135950 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 client.144143.objecter ms_dispatch 0x7f5ef0000ba0 osd_map(457..457 src has 99..457) 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 3 client.144143.objecter handle_osd_map got epochs [457,457] > 0 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 3 client.144143.objecter handle_osd_map decoding full epoch 457 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 20 client.144143.objecter dump_active .. 0 homeless 2024-09-16T03:16:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 985751744 0 0) 0x7f5ed00065d0 con 0x7f5ed0002870 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient(hunting): my global_id is 144146 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef0070ed0 con 0x7f5ed0002870 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3950401695 0 0) 0x7f5ef0070ed0 con 0x7f5ed0002870 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ed00065d0 con 0x7f5ed0002870 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ef658c640 10 monclient: get_auth_request con 0x7f5edc03e080 auth_method 0 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5ee8003980 con 0x7f5ed0002870 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 10 monclient: dump: 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2168910926 0 0) 0x7f5ed00065d0 con 0x7f5ed0002870 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.826+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ed000a170 con 0x7f5ed0002870 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.830+0000 7f5ee67fc640 10 client.144143.objecter ms_handle_connect 0x7f5edc03e080 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.830+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5ee8005bc0 con 0x7f5ed0002870 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.830+0000 7f5ee67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.830+0000 7f5ee67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.830+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 >> v1:172.21.15.45:6790/0 conn(0x7f5ed0002870 legacy=0x7f5ed0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.830+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 >> v1:172.21.15.45:6790/0 conn(0x7f5ed0002870 legacy=0x7f5ed0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.834+0000 7f5ef7fef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.834+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ed0009e30 con 0x7f5ed00090f0 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.834+0000 7f5ee67fc640 10 client.144143.objecter ms_handle_connect 0x7f5ed00090f0 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.834+0000 7f5ee67fc640 10 client.144143.objecter resend_mon_ops 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 682914609 0 0) 0x7f5ed0009e30 con 0x7f5ed00090f0 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 10 monclient(hunting): my global_id is 144149 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed00065d0 con 0x7f5ed00090f0 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3534010159 0 0) 0x7f5ed00065d0 con 0x7f5ed00090f0 2024-09-16T03:16:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ed0009e30 con 0x7f5ed00090f0 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5ee80051a0 con 0x7f5ed00090f0 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 10 monclient: dump: 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1475903230 0 0) 0x7f5ed0009e30 con 0x7f5ed00090f0 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ed000c400 con 0x7f5ed00090f0 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3966 (unknown 0 0 1892750674) 0x7f5ee8005130 con 0x7f5ed00090f0 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 >> v1:172.21.15.45:6790/0 conn(0x7f5ed00090f0 legacy=0x7f5ed0009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ee67fc640 1 -- 172.21.15.45:0/1723932260 >> v1:172.21.15.45:6790/0 conn(0x7f5ed00090f0 legacy=0x7f5ed0009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:17.853 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 46, 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 2911, 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:17.854 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:17.855 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:17.856 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:16:17.857 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:17.858 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:17.859 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:17.860 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.861 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:16:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:16:17.863 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:17.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:17.865 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:16:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:16:17.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 10 librados: watch_flush enter 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 10 librados: watch_flush exit 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 20 client.144143.objecter shutdown clearing up homeless session... 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 10 client.144143.objecter successfully canceled tick 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 >> v1:172.21.15.46:6812/149360280 conn(0x7f5edc03e080 legacy=0x7f5edc040540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 10 monclient: shutdown 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 >> v1:172.21.15.45:6790/0 conn(0x7f5ef0135950 legacy=0x7f5ef006eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef658c640 1 -- 172.21.15.45:0/1723932260 reap_dead start 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef658c640 1 -- 172.21.15.45:0/1723932260 reap_dead start 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 shutdown_connections 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 >> 172.21.15.45:0/1723932260 conn(0x7f5ef00a2bd0 msgr2=0x7f5ef013ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 shutdown_connections 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 1 -- 172.21.15.45:0/1723932260 wait complete. 2024-09-16T03:16:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.838+0000 7f5ef7fef640 1 librados: shutdown 2024-09-16T03:16:17.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:16:17.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:16:17.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:16:17.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient: get_monmap_and_config 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient: build_initial_monmap 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient: monmap: 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:17.984364+0000 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:17.984364+0000 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 Processor -- start 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 -- start start 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient: init 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0c0a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0c0a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient(hunting): picked mon.noname-c con 0x7f1c0c073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c09d9e0 con 0x7f1c0c0a5810 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c09dc20 con 0x7f1c0c073d40 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c073360 con 0x7f1c0c0a4d20 2024-09-16T03:16:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient(hunting): authenticate will time out at 1668.633982s 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0b7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1c0c0a4d20 0x7f1c0c0a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37522/0 (socket says 172.21.15.45:37522) 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0b7fe640 1 -- 172.21.15.45:0/2730352805 learned_addr learned my addr 172.21.15.45:0/2730352805 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264085765 0 0) 0x7f1c0c073360 con 0x7f1c0c0a4d20 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient(hunting): my global_id is 144152 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bec003530 con 0x7f1c0c0a4d20 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2280815991 0 0) 0x7f1c0c09d9e0 con 0x7f1c0c0a5810 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient(hunting): my global_id is 134253 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c0c073360 con 0x7f1c0c0a5810 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2142152654 0 0) 0x7f1bec003530 con 0x7f1c0c0a4d20 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 >> v1:172.21.15.46:6789/0 conn(0x7f1c0c073d40 legacy=0x7f1c0c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 >> v1:172.21.15.45:6789/0 conn(0x7f1c0c0a5810 legacy=0x7f1c0c13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 monclient: found mon.noname-b 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c09cf30 con 0x7f1c0c0a4d20 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1bfc0016a0 con 0x7f1c0c0a4d20 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: dump: 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 5 monclient: authenticate success, global_id 144152 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1bfc001840 con 0x7f1c0c0a4d20 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 1 -- 172.21.15.45:0/2730352805 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1bfc001cd0 con 0x7f1c0c0a4d20 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 4 set_mon_vals no callback set 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c0a7fc640 10 monclient: dump: 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 10 monclient: shutdown 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 -- 172.21.15.45:0/2730352805 >> v1:172.21.15.45:6790/0 conn(0x7f1c0c0a4d20 legacy=0x7f1c0c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 -- 172.21.15.45:0/2730352805 shutdown_connections 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 -- 172.21.15.45:0/2730352805 >> 172.21.15.45:0/2730352805 conn(0x7f1c0c0aa720 msgr2=0x7f1c0c0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 -- 172.21.15.45:0/2730352805 shutdown_connections 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.982+0000 7f1c12221640 1 -- 172.21.15.45:0/2730352805 wait complete. 2024-09-16T03:16:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient: build_initial_monmap 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient: monmap: 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 librados: starting msgr at 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 librados: starting objecter 2024-09-16T03:16:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 Processor -- start 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 -- start start 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 librados: setting wanted keys 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 librados: calling monclient init 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient: init 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0c073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient(hunting): picked mon.noname-c con 0x7f1c0c0a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0c0a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c1405e0 con 0x7f1c0c0a5810 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c1408c0 con 0x7f1c0c073d40 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c0a1d00 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient(hunting): authenticate will time out at 1668.637044s 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c0affd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1c0c0a4d20 0x7f1c0c1d2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37532/0 (socket says 172.21.15.45:37532) 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c0affd640 1 -- 172.21.15.45:0/1372028325 learned_addr learned my addr 172.21.15.45:0/1372028325 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.?.objecter ms_handle_connect 0x7f1c0c0a5810 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.?.objecter ms_handle_connect 0x7f1c0c0a4d20 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1787013864 0 0) 0x7f1c0c1405e0 con 0x7f1c0c0a5810 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): my global_id is 134256 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bf0004ec0 con 0x7f1c0c0a5810 2024-09-16T03:16:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1051184366 0 0) 0x7f1c0c0a1d00 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): my global_id is 144155 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c0c1405e0 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.?.objecter ms_handle_connect 0x7f1c0c073d40 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2027264610 0 0) 0x7f1c0c1405e0 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c0c0a1d00 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2941637492 0 0) 0x7f1bf0004ec0 con 0x7f1c0c0a5810 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c0c1405e0 con 0x7f1c0c0a5810 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958682835 0 0) 0x7f1c0c1408c0 con 0x7f1c0c073d40 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): my global_id is 23158 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bf0004ec0 con 0x7f1c0c073d40 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1bf40013e0 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): dump: 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1c04001bc0 con 0x7f1c0c0a5810 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): dump: 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1475835722 0 0) 0x7f1c0c0a1d00 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 >> v1:172.21.15.46:6789/0 conn(0x7f1c0c073d40 legacy=0x7f1c0c1cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 >> v1:172.21.15.45:6789/0 conn(0x7f1c0c0a5810 legacy=0x7f1c0c1d5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient: _finish_hunting 0 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 monclient: found mon.c 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c0a1510 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient: _finish_auth 0 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient: _check_auth_tickets 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 5 monclient: authenticate success, global_id 144155 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient: _renew_subs 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0c140300 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 client.144155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient: _renew_subs 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1c0c13dd50 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 librados: init done 2024-09-16T03:16:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1bf40016e0 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1bf4003a50 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient: dump: 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c09ffb640 4 set_mon_vals no callback set 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1be40065d0 con 0x7f1be4002870 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f1bf4003bf0 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(458..458 src has 99..458) ==== 5367+0+0 (unknown 825709455 0 0) 0x7f1bf404b750 con 0x7f1c0c0a4d20 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.144155.objecter ms_dispatch 0x7f1c0c000ba0 osd_map(458..458 src has 99..458) 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 3 client.144155.objecter handle_osd_map got epochs [458,458] > 0 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 3 client.144155.objecter handle_osd_map decoding full epoch 458 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 20 client.144155.objecter dump_active .. 0 homeless 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.144155.objecter ms_handle_connect 0x7f1be4002870 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.144155.objecter resend_mon_ops 2024-09-16T03:16:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c0bfff640 10 monclient: get_auth_request con 0x7f1bf003e2b0 auth_method 0 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022825608 0 0) 0x7f1be40065d0 con 0x7f1be4002870 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 monclient(hunting): my global_id is 23161 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c0c0a1d00 con 0x7f1be4002870 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.986+0000 7f1c03fff640 10 client.144155.objecter ms_handle_connect 0x7f1bf003e2b0 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1030742782 0 0) 0x7f1c0c0a1d00 con 0x7f1be4002870 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1be40065d0 con 0x7f1be4002870 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1bfc001660 con 0x7f1be4002870 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:16:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 10 monclient: dump: 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2344436318 0 0) 0x7f1be40065d0 con 0x7f1be4002870 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1be400a170 con 0x7f1be4002870 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1bfc0019a0 con 0x7f1be4002870 2024-09-16T03:16:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 >> v1:172.21.15.46:6789/0 conn(0x7f1be4002870 legacy=0x7f1be4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.990+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 >> v1:172.21.15.46:6789/0 conn(0x7f1be4002870 legacy=0x7f1be4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.994+0000 7f1c12221640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:16:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.994+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1be4009e80 con 0x7f1be4009140 2024-09-16T03:16:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 client.144155.objecter ms_handle_connect 0x7f1be4009140 2024-09-16T03:16:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 client.144155.objecter resend_mon_ops 2024-09-16T03:16:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915078286 0 0) 0x7f1be4009e80 con 0x7f1be4009140 2024-09-16T03:16:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 monclient(hunting): my global_id is 23164 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1be40065d0 con 0x7f1be4009140 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3429850902 0 0) 0x7f1be40065d0 con 0x7f1be4009140 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1be4009e80 con 0x7f1be4009140 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1bfc004af0 con 0x7f1be4009140 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 monclient: dump: 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 550763097 0 0) 0x7f1be4009e80 con 0x7f1be4009140 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1be400c420 con 0x7f1be4009140 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 706061061) 0x7f1bfc004a80 con 0x7f1be4009140 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 >> v1:172.21.15.46:6789/0 conn(0x7f1be4009140 legacy=0x7f1be40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c03fff640 1 -- 172.21.15.45:0/1372028325 >> v1:172.21.15.46:6789/0 conn(0x7f1be4009140 legacy=0x7f1be40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 46, 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 699800, 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 3, 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:16:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:16:18.026 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:16:18.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 10 librados: watch_flush enter 2024-09-16T03:16:18.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 10 librados: watch_flush exit 2024-09-16T03:16:18.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 20 client.144155.objecter shutdown clearing up homeless session... 2024-09-16T03:16:18.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 10 client.144155.objecter successfully canceled tick 2024-09-16T03:16:18.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 >> v1:172.21.15.46:6812/149360280 conn(0x7f1bf003e2b0 legacy=0x7f1bf0040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 10 monclient: shutdown 2024-09-16T03:16:18.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 >> v1:172.21.15.45:6790/0 conn(0x7f1c0c0a4d20 legacy=0x7f1c0c1d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c0bfff640 1 -- 172.21.15.45:0/1372028325 reap_dead start 2024-09-16T03:16:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c0bfff640 1 -- 172.21.15.45:0/1372028325 reap_dead start 2024-09-16T03:16:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 shutdown_connections 2024-09-16T03:16:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 >> 172.21.15.45:0/1372028325 conn(0x7f1c0c0aa720 msgr2=0x7f1c0c0a40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 shutdown_connections 2024-09-16T03:16:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 1 -- 172.21.15.45:0/1372028325 wait complete. 2024-09-16T03:16:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:17.998+0000 7f1c12221640 1 librados: shutdown 2024-09-16T03:16:18.029 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:16:18.030 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:16:18.030 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:16:18.030 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:16:18.030 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-16T03:16:18.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-16T03:16:18.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient: get_monmap_and_config 2024-09-16T03:16:18.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient: build_initial_monmap 2024-09-16T03:16:18.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient: monmap: 2024-09-16T03:16:18.143 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:18.143 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.143 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:18.144688+0000 2024-09-16T03:16:18.144 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:18.144688+0000 2024-09-16T03:16:18.144 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:18.144 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.144 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:18.144 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:18.144 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:18.144 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 Processor -- start 2024-09-16T03:16:18.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 -- start start 2024-09-16T03:16:18.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient: init 2024-09-16T03:16:18.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:18.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:16:18.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient(hunting): picked mon.noname-a con 0x7f4078135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:18.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient(hunting): picked mon.noname-b con 0x7f407809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:18.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient(hunting): picked mon.noname-c con 0x7f407813c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f407809f7e0 con 0x7f4078135950 2024-09-16T03:16:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f407809b160 con 0x7f407813c010 2024-09-16T03:16:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f407809b440 con 0x7f407809fc40 2024-09-16T03:16:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient(hunting): authenticate will time out at 1668.793901s 2024-09-16T03:16:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407ca40640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4078135950 0x7f4078135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39604/0 (socket says 172.21.15.45:39604) 2024-09-16T03:16:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407ca40640 1 -- 172.21.15.45:0/2307711095 learned_addr learned my addr 172.21.15.45:0/2307711095 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f4077fff640 1 --1- 172.21.15.45:0/2307711095 >> v1:172.21.15.45:6790/0 conn(0x7f407809fc40 0x7f407813a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37540/0 (socket says 172.21.15.45:37540) 2024-09-16T03:16:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 947216327 0 0) 0x7f407809b440 con 0x7f407809fc40 2024-09-16T03:16:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient(hunting): my global_id is 144158 2024-09-16T03:16:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4068003530 con 0x7f407809fc40 2024-09-16T03:16:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4203316897 0 0) 0x7f407809f7e0 con 0x7f4078135950 2024-09-16T03:16:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient(hunting): my global_id is 134259 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f407809b440 con 0x7f4078135950 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 892593603 0 0) 0x7f4068003530 con 0x7f407809fc40 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 >> v1:172.21.15.46:6789/0 conn(0x7f407813c010 legacy=0x7f407813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 >> v1:172.21.15.45:6789/0 conn(0x7f4078135950 legacy=0x7f4078135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: _finish_hunting 0 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 monclient: found mon.noname-b 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4078136440 con 0x7f407809fc40 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: _finish_auth 0 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: _check_auth_tickets 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f406c0016a0 con 0x7f407809fc40 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: dump: 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.159 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 5 monclient: authenticate success, global_id 144158 2024-09-16T03:16:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f406c001840 con 0x7f407809fc40 2024-09-16T03:16:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 1 -- 172.21.15.45:0/2307711095 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f406c001cd0 con 0x7f407809fc40 2024-09-16T03:16:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 4 set_mon_vals no callback set 2024-09-16T03:16:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f40777fe640 10 monclient: dump: 2024-09-16T03:16:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.161 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.161 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.161 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.161 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.161 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.161 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.161 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.161 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.161 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 10 monclient: shutdown 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 -- 172.21.15.45:0/2307711095 >> v1:172.21.15.45:6790/0 conn(0x7f407809fc40 legacy=0x7f407813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 -- 172.21.15.45:0/2307711095 shutdown_connections 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 -- 172.21.15.45:0/2307711095 >> 172.21.15.45:0/2307711095 conn(0x7f40780a2bd0 msgr2=0x7f40780a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 -- 172.21.15.45:0/2307711095 shutdown_connections 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.142+0000 7f407eca4640 1 -- 172.21.15.45:0/2307711095 wait complete. 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient: build_initial_monmap 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient: monmap: 2024-09-16T03:16:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 librados: starting msgr at 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 librados: starting objecter 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 Processor -- start 2024-09-16T03:16:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 -- start start 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 librados: setting wanted keys 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 librados: calling monclient init 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient: init 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient(hunting): picked mon.noname-c con 0x7f407809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient(hunting): picked mon.noname-b con 0x7f4078135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient(hunting): picked mon.noname-a con 0x7f407813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f407809d2c0 con 0x7f407813c010 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40780712c0 con 0x7f4078135950 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40780715a0 con 0x7f407809fc40 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient(hunting): authenticate will time out at 1668.797018s 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407d241640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f407813c010 0x7f407806f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39614/0 (socket says 172.21.15.45:39614) 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407d241640 1 -- 172.21.15.45:0/4063153793 learned_addr learned my addr 172.21.15.45:0/4063153793 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.?.objecter ms_handle_connect 0x7f407813c010 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.?.objecter ms_handle_connect 0x7f407809fc40 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3335770206 0 0) 0x7f407809d2c0 con 0x7f407813c010 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): my global_id is 134262 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f405c004ef0 con 0x7f407813c010 2024-09-16T03:16:18.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786095819 0 0) 0x7f40780715a0 con 0x7f407809fc40 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): my global_id is 144161 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f407809d2c0 con 0x7f407809fc40 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.?.objecter ms_handle_connect 0x7f4078135950 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 741015820 0 0) 0x7f405c004ef0 con 0x7f407813c010 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40780715a0 con 0x7f407813c010 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4070001d40 con 0x7f407813c010 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): dump: 2024-09-16T03:16:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 210140990 0 0) 0x7f407809d2c0 con 0x7f407809fc40 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f405c004ef0 con 0x7f407809fc40 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4064001220 con 0x7f407809fc40 2024-09-16T03:16:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): dump: 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3573123401 0 0) 0x7f40780712c0 con 0x7f4078135950 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): my global_id is 23170 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f407809d2c0 con 0x7f4078135950 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1448964163 0 0) 0x7f40780715a0 con 0x7f407813c010 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 >> v1:172.21.15.45:6790/0 conn(0x7f407809fc40 legacy=0x7f407806e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 >> v1:172.21.15.46:6789/0 conn(0x7f4078135950 legacy=0x7f407806ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: _finish_hunting 0 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 monclient: found mon.a 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4078070360 con 0x7f407813c010 2024-09-16T03:16:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: _finish_auth 0 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: _check_auth_tickets 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 5 monclient: authenticate success, global_id 134262 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient: _renew_subs 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40781d3b90 con 0x7f407813c010 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 client.134262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient: _renew_subs 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f40781d3f30 con 0x7f407813c010 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 librados: init done 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4070001f00 con 0x7f407813c010 2024-09-16T03:16:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4070003df0 con 0x7f407813c010 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f4076ffd640 4 set_mon_vals no callback set 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: dump: 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f40580065d0 con 0x7f4058002870 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4070003f90 con 0x7f407813c010 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(458..458 src has 99..458) ==== 5367+0+0 (unknown 825709455 0 0) 0x7f407004baa0 con 0x7f407813c010 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.134262.objecter ms_dispatch 0x7f4078000ba0 osd_map(458..458 src has 99..458) 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 3 client.134262.objecter handle_osd_map got epochs [458,458] > 0 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 3 client.134262.objecter handle_osd_map decoding full epoch 458 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 20 client.134262.objecter dump_active .. 0 homeless 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.134262.objecter ms_handle_connect 0x7f4058002870 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.134262.objecter resend_mon_ops 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 239578290 0 0) 0x7f40580065d0 con 0x7f4058002870 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient(hunting): my global_id is 23173 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40780715a0 con 0x7f4058002870 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f4077fff640 10 monclient: get_auth_request con 0x7f405c03e340 auth_method 0 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 client.134262.objecter ms_handle_connect 0x7f405c03e340 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 319214679 0 0) 0x7f40780715a0 con 0x7f4058002870 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40580065d0 con 0x7f4058002870 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4064004990 con 0x7f4058002870 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.146+0000 7f40757fa640 10 monclient: dump: 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.150+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2255247911 0 0) 0x7f40580065d0 con 0x7f4058002870 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.150+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f405800a170 con 0x7f4058002870 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.150+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4064004990 con 0x7f4058002870 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.150+0000 7f40757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.150+0000 7f40757fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.150+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 >> v1:172.21.15.46:6789/0 conn(0x7f4058002870 legacy=0x7f4058004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.150+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 >> v1:172.21.15.46:6789/0 conn(0x7f4058002870 legacy=0x7f4058004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:16:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4058009ea0 con 0x7f4058009160 2024-09-16T03:16:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 client.134262.objecter ms_handle_connect 0x7f4058009160 2024-09-16T03:16:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 client.134262.objecter resend_mon_ops 2024-09-16T03:16:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2848394503 0 0) 0x7f4058009ea0 con 0x7f4058009160 2024-09-16T03:16:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 monclient(hunting): my global_id is 23176 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40580065d0 con 0x7f4058009160 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3559235898 0 0) 0x7f40580065d0 con 0x7f4058009160 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4058009ea0 con 0x7f4058009160 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f40640050a0 con 0x7f4058009160 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 monclient: dump: 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 604777787 0 0) 0x7f4058009ea0 con 0x7f4058009160 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f405800c400 con 0x7f4058009160 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (unknown 0 0 2946029456) 0x7f4064006570 con 0x7f4058009160 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:16:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 >> v1:172.21.15.46:6789/0 conn(0x7f4058009160 legacy=0x7f40580095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f40757fa640 1 -- 172.21.15.45:0/4063153793 >> v1:172.21.15.46:6789/0 conn(0x7f4058009160 legacy=0x7f40580095f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:16:18.179 INFO:teuthology.orchestra.run.smithi045.stdout: "ret": 0, 2024-09-16T03:16:18.179 INFO:teuthology.orchestra.run.smithi045.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T03:16:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 10 librados: watch_flush enter 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 10 librados: watch_flush exit 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 20 client.134262.objecter shutdown clearing up homeless session... 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 10 client.134262.objecter successfully canceled tick 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 >> v1:172.21.15.46:6812/149360280 conn(0x7f405c03e340 legacy=0x7f405c040800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 10 monclient: shutdown 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 >> v1:172.21.15.45:6789/0 conn(0x7f407813c010 legacy=0x7f407806f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407d241640 1 -- 172.21.15.45:0/4063153793 reap_dead start 2024-09-16T03:16:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407d241640 1 -- 172.21.15.45:0/4063153793 reap_dead start 2024-09-16T03:16:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 shutdown_connections 2024-09-16T03:16:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 >> 172.21.15.45:0/4063153793 conn(0x7f40780a2bd0 msgr2=0x7f407813eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 shutdown_connections 2024-09-16T03:16:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 1 -- 172.21.15.45:0/4063153793 wait complete. 2024-09-16T03:16:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.158+0000 7f407eca4640 1 librados: shutdown 2024-09-16T03:16:18.182 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:16:18.182 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:16:18.182 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:16:18.221 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:16:18.222 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:16:18.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:16:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient: get_monmap_and_config 2024-09-16T03:16:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient: build_initial_monmap 2024-09-16T03:16:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient: monmap: 2024-09-16T03:16:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:18.333294+0000 2024-09-16T03:16:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:18.333294+0000 2024-09-16T03:16:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:18.332 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 1 Processor -- start 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 1 -- start start 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient: init 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient(hunting): picked mon.noname-b con 0x7f4f740a25d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7409a850 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient(hunting): picked mon.noname-a con 0x7f4f7409ac60 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7409c3f0 con 0x7f4f7409ac60 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7409c6d0 con 0x7f4f7409a850 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7409c9b0 con 0x7f4f740a25d0 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 10 monclient(hunting): authenticate will time out at 1668.983129s 2024-09-16T03:16:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f737fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4f740a25d0 0x7f4f7409a140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37586/0 (socket says 172.21.15.45:37586) 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f737fe640 1 -- 172.21.15.45:0/3072548341 learned_addr learned my addr 172.21.15.45:0/3072548341 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f72ffd640 1 -- 172.21.15.45:0/3072548341 >> v1:172.21.15.46:6789/0 conn(0x7f4f7409a850 legacy=0x7f4f74139250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3571967636 0 0) 0x7f4f7409c9b0 con 0x7f4f740a25d0 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient(hunting): my global_id is 144164 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f5c003530 con 0x7f4f740a25d0 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1000596466 0 0) 0x7f4f7409c3f0 con 0x7f4f7409ac60 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient(hunting): my global_id is 134265 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f7409c9b0 con 0x7f4f7409ac60 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2757282661 0 0) 0x7f4f5c003530 con 0x7f4f740a25d0 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 >> v1:172.21.15.46:6789/0 conn(0x7f4f7409a850 legacy=0x7f4f74139250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 >> v1:172.21.15.45:6789/0 conn(0x7f4f7409ac60 legacy=0x7f4f7413c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient: _finish_hunting 0 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 monclient: found mon.noname-b 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7409b350 con 0x7f4f740a25d0 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient: _finish_auth 0 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient: _check_auth_tickets 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4f64001950 con 0x7f4f740a25d0 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 5 monclient: authenticate success, global_id 144164 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f727fc640 10 monclient: dump: 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.330+0000 7f4f79c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f64001af0 con 0x7f4f740a25d0 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f727fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f727fc640 1 -- 172.21.15.45:0/3072548341 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4f64001ec0 con 0x7f4f740a25d0 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 4 set_mon_vals no callback set 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f727fc640 10 monclient: dump: 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: shutdown 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- 172.21.15.45:0/3072548341 >> v1:172.21.15.45:6790/0 conn(0x7f4f740a25d0 legacy=0x7f4f7409a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- 172.21.15.45:0/3072548341 shutdown_connections 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- 172.21.15.45:0/3072548341 >> 172.21.15.45:0/3072548341 conn(0x7f4f740a2bd0 msgr2=0x7f4f740a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- 172.21.15.45:0/3072548341 shutdown_connections 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- 172.21.15.45:0/3072548341 wait complete. 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: build_initial_monmap 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: monmap: 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:18.343 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 librados: starting msgr at 2024-09-16T03:16:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 librados: starting objecter 2024-09-16T03:16:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 Processor -- start 2024-09-16T03:16:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- start start 2024-09-16T03:16:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 librados: setting wanted keys 2024-09-16T03:16:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 librados: calling monclient init 2024-09-16T03:16:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: init 2024-09-16T03:16:18.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7409a850 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7409ac60 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient(hunting): picked mon.noname-a con 0x7f4f740a25d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f74097680 con 0x7f4f740a25d0 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7409a5b0 con 0x7f4f7409a850 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7409d2c0 con 0x7f4f7409ac60 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient(hunting): authenticate will time out at 1668.986264s 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f737fe640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f4f7409a850 legacy=0x7f4f741c9200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f72ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4f7409ac60 0x7f4f741c9910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37600/0 (socket says 172.21.15.45:37600) 2024-09-16T03:16:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f73fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4f740a25d0 0x7f4f741d4990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39632/0 (socket says 172.21.15.45:39632) 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f72ffd640 1 -- 172.21.15.45:0/2128526416 learned_addr learned my addr 172.21.15.45:0/2128526416 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 client.?.objecter ms_handle_connect 0x7f4f7409ac60 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 client.?.objecter ms_handle_connect 0x7f4f740a25d0 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3151571924 0 0) 0x7f4f7409d2c0 con 0x7f4f7409ac60 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): my global_id is 144167 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f60004ef0 con 0x7f4f7409ac60 2024-09-16T03:16:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404878181 0 0) 0x7f4f6c001900 con 0x7f4f740a25d0 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): my global_id is 134268 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f7409d2c0 con 0x7f4f740a25d0 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 470484741 0 0) 0x7f4f60004ef0 con 0x7f4f7409ac60 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f6c001900 con 0x7f4f7409ac60 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1675114410 0 0) 0x7f4f7409d2c0 con 0x7f4f740a25d0 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f60004ef0 con 0x7f4f740a25d0 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4f68004260 con 0x7f4f7409ac60 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): dump: 2024-09-16T03:16:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4f6c0013c0 con 0x7f4f740a25d0 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient(hunting): dump: 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1605253202 0 0) 0x7f4f6c001900 con 0x7f4f7409ac60 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 >> v1:172.21.15.46:6789/0 conn(0x7f4f7409a850 legacy=0x7f4f741c9200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 >> v1:172.21.15.45:6789/0 conn(0x7f4f740a25d0 legacy=0x7f4f741d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 monclient: found mon.c 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f74099f30 con 0x7f4f7409ac60 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient: _finish_auth 0 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 5 monclient: authenticate success, global_id 144167 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: _renew_subs 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7406fbb0 con 0x7f4f7409ac60 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 client.144167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: _renew_subs 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f7406f5c0 con 0x7f4f7409ac60 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f79c50640 1 librados: init done 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f68004400 con 0x7f4f7409ac60 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4f68004850 con 0x7f4f7409ac60 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f71ffb640 4 set_mon_vals no callback set 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.334+0000 7f4f5bfff640 10 monclient: dump: 2024-09-16T03:16:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f79c50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f79c50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f79c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f54005090 con 0x7f4f7409ac60 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4f680049f0 con 0x7f4f7409ac60 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(458..458 src has 99..458) ==== 5367+0+0 (unknown 825709455 0 0) 0x7f4f6804b7d0 con 0x7f4f7409ac60 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f5bfff640 10 client.144167.objecter ms_dispatch 0x7f4f74000ba0 osd_map(458..458 src has 99..458) 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f5bfff640 3 client.144167.objecter handle_osd_map got epochs [458,458] > 0 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f5bfff640 3 client.144167.objecter handle_osd_map decoding full epoch 458 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f5bfff640 20 client.144167.objecter dump_active .. 0 homeless 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f737fe640 10 monclient: get_auth_request con 0x7f4f6003e010 auth_method 0 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.338+0000 7f4f5bfff640 10 client.144167.objecter ms_handle_connect 0x7f4f6003e010 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.342+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4f68016ac0 con 0x7f4f7409ac60 2024-09-16T03:16:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.342+0000 7f4f5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.342+0000 7f4f5bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:18.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.574+0000 7f4f79c50640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:16:18.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.574+0000 7f4f79c50640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:18.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.574+0000 7f4f79c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:18.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.574+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f54005380 con 0x7f4f7409ac60 2024-09-16T03:16:18.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.574+0000 7f4f5bfff640 1 -- 172.21.15.45:0/2128526416 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 3024189506) 0x7f4f680163c0 con 0x7f4f7409ac60 2024-09-16T03:16:18.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.574+0000 7f4f5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:18.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.574+0000 7f4f5bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:18.578 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 10 librados: watch_flush enter 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 10 librados: watch_flush exit 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 20 client.144167.objecter shutdown clearing up homeless session... 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 10 client.144167.objecter successfully canceled tick 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 >> v1:172.21.15.46:6812/149360280 conn(0x7f4f6003e010 legacy=0x7f4f600404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 10 monclient: shutdown 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 >> v1:172.21.15.45:6790/0 conn(0x7f4f7409ac60 legacy=0x7f4f741c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 shutdown_connections 2024-09-16T03:16:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 >> 172.21.15.45:0/2128526416 conn(0x7f4f740a2bd0 msgr2=0x7f4f740a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 shutdown_connections 2024-09-16T03:16:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 1 -- 172.21.15.45:0/2128526416 wait complete. 2024-09-16T03:16:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:18.578+0000 7f4f79c50640 1 librados: shutdown 2024-09-16T03:16:21.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:16:21.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient: get_monmap_and_config 2024-09-16T03:16:21.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient: build_initial_monmap 2024-09-16T03:16:21.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient: monmap: 2024-09-16T03:16:21.728 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:21.729 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:21.729 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:21.729833+0000 2024-09-16T03:16:21.729 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:21.729833+0000 2024-09-16T03:16:21.729 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:21.729 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:21.729 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:21.729 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:21.729 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:21.729 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:21.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 1 Processor -- start 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 1 -- start start 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient: init 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient(hunting): picked mon.noname-a con 0x7f959c09b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient(hunting): picked mon.noname-b con 0x7f959c09b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient(hunting): picked mon.noname-c con 0x7f959c13b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959c09cc20 con 0x7f959c09b8e0 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959c09d820 con 0x7f959c13b000 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959c09b6a0 con 0x7f959c09b250 2024-09-16T03:16:21.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:21.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a3034640 10 monclient(hunting): authenticate will time out at 1672.379687s 2024-09-16T03:16:21.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a15d1640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f959c13b000 legacy=0x7f959c13d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:21.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f959bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f959c09b250 0x7f959c1398e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46762/0 (socket says 172.21.15.45:46762) 2024-09-16T03:16:21.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f959bfff640 1 -- 172.21.15.45:0/875945962 learned_addr learned my addr 172.21.15.45:0/875945962 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:21.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.726+0000 7f95a0dd0640 1 --1- 172.21.15.45:0/875945962 >> v1:172.21.15.45:6789/0 conn(0x7f959c09b8e0 0x7f959c09bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56000/0 (socket says 172.21.15.45:56000) 2024-09-16T03:16:21.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 -- 172.21.15.45:0/875945962 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3349054132 0 0) 0x7f959c09cc20 con 0x7f959c09b8e0 2024-09-16T03:16:21.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:16:21.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient(hunting): my global_id is 134271 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 -- 172.21.15.45:0/875945962 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9584003530 con 0x7f959c09b8e0 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 -- 172.21.15.45:0/875945962 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 298312632 0 0) 0x7f9584003530 con 0x7f959c09b8e0 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 -- 172.21.15.45:0/875945962 >> v1:172.21.15.45:6790/0 conn(0x7f959c09b250 legacy=0x7f959c1398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 -- 172.21.15.45:0/875945962 >> v1:172.21.15.46:6789/0 conn(0x7f959c13b000 legacy=0x7f959c13d3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 -- 172.21.15.45:0/875945962 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959c0a21f0 con 0x7f959c09b8e0 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:16:21.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 -- 172.21.15.45:0/875945962 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f958c0016a0 con 0x7f959c09b8e0 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 5 monclient: authenticate success, global_id 134271 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: dump: 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:21.735 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 -- 172.21.15.45:0/875945962 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f958c0018c0 con 0x7f959c09b8e0 2024-09-16T03:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:21.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 1 -- 172.21.15.45:0/875945962 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f958c001cd0 con 0x7f959c09b8e0 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 4 set_mon_vals no callback set 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959b7fe640 10 monclient: dump: 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: shutdown 2024-09-16T03:16:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- 172.21.15.45:0/875945962 >> v1:172.21.15.45:6789/0 conn(0x7f959c09b8e0 legacy=0x7f959c09bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- 172.21.15.45:0/875945962 shutdown_connections 2024-09-16T03:16:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- 172.21.15.45:0/875945962 >> 172.21.15.45:0/875945962 conn(0x7f959c0a2bd0 msgr2=0x7f959c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- 172.21.15.45:0/875945962 shutdown_connections 2024-09-16T03:16:21.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- 172.21.15.45:0/875945962 wait complete. 2024-09-16T03:16:21.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: build_initial_monmap 2024-09-16T03:16:21.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: monmap: 2024-09-16T03:16:21.742 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 librados: starting msgr at 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 librados: starting objecter 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 Processor -- start 2024-09-16T03:16:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- start start 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 librados: setting wanted keys 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 librados: calling monclient init 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: init 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient(hunting): picked mon.noname-a con 0x7f959c09b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient(hunting): picked mon.noname-b con 0x7f959c13b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient(hunting): picked mon.noname-c con 0x7f959c1ce7b0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959c0a5850 con 0x7f959c09b250 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959c12ccf0 con 0x7f959c13b000 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959c12cfd0 con 0x7f959c1ce7b0 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:21.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient(hunting): authenticate will time out at 1672.383005s 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a0dd0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f959c09b250 0x7f959c072530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56002/0 (socket says 172.21.15.45:56002) 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a0dd0640 1 -- 172.21.15.45:0/2333977596 learned_addr learned my addr 172.21.15.45:0/2333977596 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f959bfff640 1 -- 172.21.15.45:0/2333977596 >> v1:172.21.15.46:6789/0 conn(0x7f959c13b000 legacy=0x7f959c072c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 client.?.objecter ms_handle_connect 0x7f959c09b250 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 client.?.objecter ms_handle_connect 0x7f959c1ce7b0 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3672008120 0 0) 0x7f959c0a21f0 con 0x7f959c09b250 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient(hunting): my global_id is 134274 2024-09-16T03:16:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9588004ef0 con 0x7f959c09b250 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1142771232 0 0) 0x7f9588004ef0 con 0x7f959c09b250 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f959c0a21f0 con 0x7f959c09b250 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f958c001c50 con 0x7f959c09b250 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient(hunting): dump: 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:21.746 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1691076264 0 0) 0x7f959c0a21f0 con 0x7f959c09b250 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 >> v1:172.21.15.45:6790/0 conn(0x7f959c1ce7b0 legacy=0x7f959c1d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 >> v1:172.21.15.46:6789/0 conn(0x7f959c13b000 legacy=0x7f959c072c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient: _finish_hunting 0 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 monclient: found mon.a 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959c073350 con 0x7f959c09b250 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient: _finish_auth 0 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 10 monclient: _check_auth_tickets 2024-09-16T03:16:21.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 5 monclient: authenticate success, global_id 134274 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: _renew_subs 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f959c136dd0 con 0x7f959c09b250 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 client.134274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: _renew_subs 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f959c12c730 con 0x7f959c09b250 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95a3034640 1 librados: init done 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.730+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f958c001820 con 0x7f959c09b250 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f958c004d20 con 0x7f959c09b250 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:21.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:21.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f959a7fc640 4 set_mon_vals no callback set 2024-09-16T03:16:21.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 10 monclient: dump: 2024-09-16T03:16:21.749 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:21.749 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:21.749 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:21.749 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:21.749 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:21.749 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:21.749 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95a3034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95a3034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95a3034640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f957c005090 con 0x7f959c09b250 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f958c004ec0 con 0x7f959c09b250 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(458..458 src has 99..458) ==== 5367+0+0 (unknown 825709455 0 0) 0x7f958c04b820 con 0x7f959c09b250 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 10 client.134274.objecter ms_dispatch 0x7f959c000ba0 osd_map(458..458 src has 99..458) 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 3 client.134274.objecter handle_osd_map got epochs [458,458] > 0 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 3 client.134274.objecter handle_osd_map decoding full epoch 458 2024-09-16T03:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 20 client.134274.objecter dump_active .. 0 homeless 2024-09-16T03:16:21.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f959bfff640 10 monclient: get_auth_request con 0x7f958803dfa0 auth_method 0 2024-09-16T03:16:21.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.734+0000 7f95997fa640 10 client.134274.objecter ms_handle_connect 0x7f958803dfa0 2024-09-16T03:16:21.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.738+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f959c0a5850 con 0x7f959c09b250 2024-09-16T03:16:21.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.738+0000 7f95997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:21.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.738+0000 7f95997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:21.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.974+0000 7f95a3034640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.974+0000 7f95a3034640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.974+0000 7f95a3034640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.974+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f957c005380 con 0x7f959c09b250 2024-09-16T03:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.974+0000 7f95997fa640 1 -- 172.21.15.45:0/2333977596 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2336476482) 0x7f958c016b70 con 0x7f959c09b250 2024-09-16T03:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.974+0000 7f95997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.974+0000 7f95997fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:21.977 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:16:21.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 10 librados: watch_flush enter 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 10 librados: watch_flush exit 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 20 client.134274.objecter shutdown clearing up homeless session... 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 10 client.134274.objecter successfully canceled tick 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 >> v1:172.21.15.46:6812/149360280 conn(0x7f958803dfa0 legacy=0x7f9588040460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 10 monclient: shutdown 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 >> v1:172.21.15.45:6789/0 conn(0x7f959c09b250 legacy=0x7f959c072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 shutdown_connections 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 >> 172.21.15.45:0/2333977596 conn(0x7f959c0a2bd0 msgr2=0x7f959c0a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 shutdown_connections 2024-09-16T03:16:21.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 1 -- 172.21.15.45:0/2333977596 wait complete. 2024-09-16T03:16:21.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:21.978+0000 7f95a3034640 1 librados: shutdown 2024-09-16T03:16:24.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:16:25.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient: get_monmap_and_config 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient: build_initial_monmap 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient: monmap: 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:25.130487+0000 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:25.130487+0000 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:25.130 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:25.130 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:25.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 1 Processor -- start 2024-09-16T03:16:25.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 1 -- start start 2024-09-16T03:16:25.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient: init 2024-09-16T03:16:25.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:25.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:16:25.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient(hunting): picked mon.noname-a con 0x7fa5bc09c570 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:25.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fa5bc134320 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:25.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient(hunting): picked mon.noname-b con 0x7fa5bc1347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:25.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc135150 con 0x7fa5bc09c570 2024-09-16T03:16:25.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc074470 con 0x7fa5bc134320 2024-09-16T03:16:25.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc074750 con 0x7fa5bc1347d0 2024-09-16T03:16:25.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:25.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5c1bfd640 10 monclient(hunting): authenticate will time out at 1675.779855s 2024-09-16T03:16:25.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.126+0000 7fa5bbfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fa5bc1347d0 0x7fa5bc1402a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46780/0 (socket says 172.21.15.45:46780) 2024-09-16T03:16:25.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.130+0000 7fa5bbfff640 1 -- 172.21.15.45:0/2009026326 learned_addr learned my addr 172.21.15.45:0/2009026326 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:25.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.130+0000 7fa5baffd640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa5bc13cb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:25.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.334+0000 7fa5baffd640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa5bc13cb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:25.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:25.738+0000 7fa5baffd640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa5bc13cb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:26.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:26.542+0000 7fa5baffd640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa5bc13cb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:28.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.126+0000 7fa5b97fa640 10 monclient(hunting): tick 2024-09-16T03:16:28.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.126+0000 7fa5b97fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:16:28.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.126+0000 7fa5b97fa640 1 monclient(hunting): continuing hunt 2024-09-16T03:16:28.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.126+0000 7fa5b97fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:16:28.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.126+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.45:6790/0 conn(0x7fa5bc1347d0 legacy=0x7fa5bc1402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:28.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.126+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa5bc13cb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:16:28.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.45:6789/0 conn(0x7fa5bc09c570 legacy=0x7fa5bc09c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:28.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5b97fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:16:28.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5b97fa640 10 monclient(hunting): picked mon.noname-a con 0x7fa59c0036d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:28.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5b97fa640 10 monclient(hunting): picked mon.noname-c con 0x7fa5bc134320 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:28.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5b97fa640 10 monclient(hunting): picked mon.noname-b con 0x7fa5bc1347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:28.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc074a30 con 0x7fa59c0036d0 2024-09-16T03:16:28.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5baffd640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa59c0093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:28.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc074470 con 0x7fa5bc134320 2024-09-16T03:16:28.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59c0103d0 con 0x7fa5bc1347d0 2024-09-16T03:16:28.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.130+0000 7fa5b97fa640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:28.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.334+0000 7fa5baffd640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa59c0093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:28.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:28.738+0000 7fa5baffd640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa59c0093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:29.542+0000 7fa5baffd640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa59c0093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 10 monclient(hunting): tick 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 1 monclient(hunting): continuing hunt 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.45:6790/0 conn(0x7fa5bc1347d0 legacy=0x7fa59c00cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa5bc134320 legacy=0x7fa59c0093d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.45:6789/0 conn(0x7fa59c0036d0 legacy=0x7fa59c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 10 monclient(hunting): picked mon.noname-c con 0x7fa59c0106a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 10 monclient(hunting): picked mon.noname-a con 0x7fa59c009b00 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 10 monclient(hunting): picked mon.noname-b con 0x7fa5bc1347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59c0116d0 con 0x7fa59c009b00 2024-09-16T03:16:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc074470 con 0x7fa59c0106a0 2024-09-16T03:16:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 1 -- 172.21.15.45:0/2009026326 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59c009ee0 con 0x7fa5bc1347d0 2024-09-16T03:16:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5b97fa640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.130+0000 7fa5bb7fe640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa59c0106a0 legacy=0x7fa59c010ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:31.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.334+0000 7fa5bb7fe640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa59c0106a0 legacy=0x7fa59c010ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:31.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:31.738+0000 7fa5bb7fe640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa59c0106a0 legacy=0x7fa59c010ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:32.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.542+0000 7fa5bb7fe640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa59c0106a0 legacy=0x7fa59c010ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 -- 172.21.15.45:0/2009026326 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252809079 0 0) 0x7fa59c009ee0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:16:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient(hunting): my global_id is 144170 2024-09-16T03:16:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 -- 172.21.15.45:0/2009026326 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5a8003530 con 0x7fa5bc1347d0 2024-09-16T03:16:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 -- 172.21.15.45:0/2009026326 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3663844057 0 0) 0x7fa5a8003530 con 0x7fa5bc1347d0 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.46:6789/0 conn(0x7fa59c0106a0 legacy=0x7fa59c010ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.45:6789/0 conn(0x7fa59c009b00 legacy=0x7fa59c0093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 monclient: found mon.noname-b 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 -- 172.21.15.45:0/2009026326 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59c00a100 con 0x7fa5bc1347d0 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: _finish_auth 0 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 -- 172.21.15.45:0/2009026326 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa5b40015e0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 5 monclient: authenticate success, global_id 144170 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: dump: 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 -- 172.21.15.45:0/2009026326 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5b4001780 con 0x7fa5bc1347d0 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 1 -- 172.21.15.45:0/2009026326 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa5b4001b90 con 0x7fa5bc1347d0 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 4 set_mon_vals no callback set 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5ba7fc640 10 monclient: dump: 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:32.965 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:32.965 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:32.965 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:32.965 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:32.965 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:32.965 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 10 monclient: shutdown 2024-09-16T03:16:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/2009026326 >> v1:172.21.15.45:6790/0 conn(0x7fa5bc1347d0 legacy=0x7fa59c00cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/2009026326 shutdown_connections 2024-09-16T03:16:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.958+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/2009026326 >> 172.21.15.45:0/2009026326 conn(0x7fa5bc0a2bd0 msgr2=0x7fa5bc0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/2009026326 shutdown_connections 2024-09-16T03:16:32.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/2009026326 wait complete. 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient: build_initial_monmap 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient: monmap: 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:32.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 librados: starting msgr at 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 librados: starting objecter 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 Processor -- start 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 -- start start 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 librados: setting wanted keys 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 librados: calling monclient init 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient: init 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient(hunting): picked mon.noname-b con 0x7fa59c009b00 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient(hunting): picked mon.noname-a con 0x7fa59c0106a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fa5bc1347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc09bf00 con 0x7fa59c0106a0 2024-09-16T03:16:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc1336e0 con 0x7fa59c009b00 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc08e1d0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient(hunting): authenticate will time out at 1683.614641s 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5bb7fe640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7fa59c009b00 legacy=0x7fa5bc1308b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5baffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa59c0106a0 0x7fa5bc130fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45638/0 (socket says 172.21.15.45:45638) 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5bbfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fa5bc1347d0 0x7fa5bc1d5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43472/0 (socket says 172.21.15.45:43472) 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5baffd640 1 -- 172.21.15.45:0/1129158947 learned_addr learned my addr 172.21.15.45:0/1129158947 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 client.?.objecter ms_handle_connect 0x7fa5bc1347d0 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 client.?.objecter ms_handle_connect 0x7fa59c0106a0 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3560600140 0 0) 0x7fa5bc08e1d0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient(hunting): my global_id is 144173 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa59c009ee0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 616669015 0 0) 0x7fa59c009ee0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5bc08e1d0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa5b4001ad0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient(hunting): dump: 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:32.973 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 493689948 0 0) 0x7fa5bc08e1d0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 >> v1:172.21.15.46:6789/0 conn(0x7fa59c009b00 legacy=0x7fa5bc1308b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 >> v1:172.21.15.45:6789/0 conn(0x7fa59c0106a0 legacy=0x7fa5bc130fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 monclient: found mon.c 2024-09-16T03:16:32.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5bc133cb0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient: _finish_auth 0 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 5 monclient: authenticate success, global_id 144173 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient: _renew_subs 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa5bc132270 con 0x7fa5bc1347d0 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 client.144173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient: _renew_subs 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.962+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa5bc1318a0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5c1bfd640 1 librados: init done 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5b4001720 con 0x7fa5bc1347d0 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fa5b4004aa0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 10 monclient: dump: 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:32.976 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b9ffb640 4 set_mon_vals no callback set 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5c1bfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5c1bfd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5c1bfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fa5b4010b10 con 0x7fa5bc1347d0 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a0005090 con 0x7fa5bc1347d0 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(459..459 src has 99..459) ==== 5367+0+0 (unknown 1657194677 0 0) 0x7fa5b404b450 con 0x7fa5bc1347d0 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 10 client.144173.objecter ms_dispatch 0x7fa5bc000ba0 osd_map(459..459 src has 99..459) 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 3 client.144173.objecter handle_osd_map got epochs [459,459] > 0 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 3 client.144173.objecter handle_osd_map decoding full epoch 459 2024-09-16T03:16:32.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 20 client.144173.objecter dump_active .. 0 homeless 2024-09-16T03:16:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5bb7fe640 10 monclient: get_auth_request con 0x7fa59c03f6d0 auth_method 0 2024-09-16T03:16:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.966+0000 7fa5b3fff640 10 client.144173.objecter ms_handle_connect 0x7fa59c03f6d0 2024-09-16T03:16:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.970+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fa5b401f9a0 con 0x7fa5bc1347d0 2024-09-16T03:16:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.970+0000 7fa5b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:32.970+0000 7fa5b3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:33.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.214+0000 7fa5c1bfd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:16:33.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.214+0000 7fa5c1bfd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:33.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.214+0000 7fa5c1bfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:16:33.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.214+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa5a0005380 con 0x7fa5bc1347d0 2024-09-16T03:16:33.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.214+0000 7fa5b3fff640 1 -- 172.21.15.45:0/1129158947 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1502250603) 0x7fa5b4037620 con 0x7fa5bc1347d0 2024-09-16T03:16:33.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.214+0000 7fa5b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:16:33.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.214+0000 7fa5b3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:33.218 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":50,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:16:33.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 10 librados: watch_flush enter 2024-09-16T03:16:33.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 10 librados: watch_flush exit 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 20 client.144173.objecter shutdown clearing up homeless session... 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 10 client.144173.objecter successfully canceled tick 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 >> v1:172.21.15.46:6812/149360280 conn(0x7fa59c03f6d0 legacy=0x7fa59c041b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 10 monclient: shutdown 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 >> v1:172.21.15.45:6790/0 conn(0x7fa5bc1347d0 legacy=0x7fa5bc1d5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 shutdown_connections 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 >> 172.21.15.45:0/1129158947 conn(0x7fa5bc0a2bd0 msgr2=0x7fa5bc09f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 shutdown_connections 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 1 -- 172.21.15.45:0/1129158947 wait complete. 2024-09-16T03:16:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.218+0000 7fa5c1bfd640 1 librados: shutdown 2024-09-16T03:16:33.236 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:16:33.237 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:16:33.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:16:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.362+0000 7fb32a899640 10 monclient: get_monmap_and_config 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.362+0000 7fb32a899640 10 monclient: build_initial_monmap 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.362+0000 7fb32a899640 10 monclient: monmap: 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:33.367526+0000 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:33.367526+0000 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:33.367 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 Processor -- start 2024-09-16T03:16:33.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- start start 2024-09-16T03:16:33.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: init 2024-09-16T03:16:33.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:33.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): picked mon.noname-c con 0x7fb324135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): picked mon.noname-b con 0x7fb32409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): picked mon.noname-a con 0x7fb32413c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32409f7e0 con 0x7fb32413c010 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32409b160 con 0x7fb324135950 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32409b440 con 0x7fb32409fc40 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): authenticate will time out at 1684.016805s 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb323fff640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7fb324135950 legacy=0x7fb324135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb3237fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb32409fc40 0x7fb32413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43482/0 (socket says 172.21.15.45:43482) 2024-09-16T03:16:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb3237fe640 1 -- 172.21.15.45:0/3420122934 learned_addr learned my addr 172.21.15.45:0/3420122934 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb328e36640 1 --1- 172.21.15.45:0/3420122934 >> v1:172.21.15.45:6789/0 conn(0x7fb32413c010 0x7fb32413e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45644/0 (socket says 172.21.15.45:45644) 2024-09-16T03:16:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1882249006 0 0) 0x7fb32409b440 con 0x7fb32409fc40 2024-09-16T03:16:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient(hunting): my global_id is 144176 2024-09-16T03:16:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb30c003530 con 0x7fb32409fc40 2024-09-16T03:16:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2415896444 0 0) 0x7fb32409f7e0 con 0x7fb32413c010 2024-09-16T03:16:33.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient(hunting): my global_id is 134277 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb32409b440 con 0x7fb32413c010 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1357616081 0 0) 0x7fb32409b440 con 0x7fb32413c010 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 >> v1:172.21.15.45:6790/0 conn(0x7fb32409fc40 legacy=0x7fb32413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 >> v1:172.21.15.46:6789/0 conn(0x7fb324135950 legacy=0x7fb324135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 monclient: found mon.noname-a 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb324136440 con 0x7fb32413c010 2024-09-16T03:16:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: _finish_auth 0 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb31c001a00 con 0x7fb32413c010 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: dump: 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 5 monclient: authenticate success, global_id 134277 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb31c001ba0 con 0x7fb32413c010 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 1 -- 172.21.15.45:0/3420122934 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb31c001ff0 con 0x7fb32413c010 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb322ffd640 10 monclient: dump: 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 4 set_mon_vals no callback set 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: shutdown 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- 172.21.15.45:0/3420122934 >> v1:172.21.15.45:6789/0 conn(0x7fb32413c010 legacy=0x7fb32413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- 172.21.15.45:0/3420122934 shutdown_connections 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- 172.21.15.45:0/3420122934 >> 172.21.15.45:0/3420122934 conn(0x7fb3240a2bd0 msgr2=0x7fb3240a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- 172.21.15.45:0/3420122934 shutdown_connections 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- 172.21.15.45:0/3420122934 wait complete. 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: build_initial_monmap 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: monmap: 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 librados: starting msgr at 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 librados: starting objecter 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 Processor -- start 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- start start 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 librados: setting wanted keys 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 librados: calling monclient init 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: init 2024-09-16T03:16:33.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): picked mon.noname-b con 0x7fb32409fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): picked mon.noname-a con 0x7fb324135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): picked mon.noname-c con 0x7fb32413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32409d610 con 0x7fb324135950 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32409ecd0 con 0x7fb32409fc40 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32409e750 con 0x7fb32413c010 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.366+0000 7fb32a899640 10 monclient(hunting): authenticate will time out at 1684.019985s 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb3237fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb324135950 0x7fb3241d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45652/0 (socket says 172.21.15.45:45652) 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb328e36640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb32413c010 0x7fb3241d56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43498/0 (socket says 172.21.15.45:43498) 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb3237fe640 1 -- 172.21.15.45:0/4204902765 learned_addr learned my addr 172.21.15.45:0/4204902765 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb323fff640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7fb32409fc40 legacy=0x7fb324099290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:33.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 client.?.objecter ms_handle_connect 0x7fb324135950 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 client.?.objecter ms_handle_connect 0x7fb32413c010 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1656493254 0 0) 0x7fb32409d610 con 0x7fb324135950 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): my global_id is 134280 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb310004ec0 con 0x7fb324135950 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774213964 0 0) 0x7fb32409e750 con 0x7fb32413c010 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): my global_id is 144179 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb32409d610 con 0x7fb32413c010 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1619742225 0 0) 0x7fb310004ec0 con 0x7fb324135950 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb32409e750 con 0x7fb324135950 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb318001880 con 0x7fb324135950 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): dump: 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 668861001 0 0) 0x7fb32409d610 con 0x7fb32413c010 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb310004ec0 con 0x7fb32413c010 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb31c001ef0 con 0x7fb32413c010 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:33.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): dump: 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3707469101 0 0) 0x7fb32409e750 con 0x7fb324135950 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 >> v1:172.21.15.45:6790/0 conn(0x7fb32413c010 legacy=0x7fb3241d56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 >> v1:172.21.15.46:6789/0 conn(0x7fb32409fc40 legacy=0x7fb324099290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 monclient: found mon.a 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32409efb0 con 0x7fb324135950 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: _finish_auth 0 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 5 monclient: authenticate success, global_id 134280 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 10 monclient: _renew_subs 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:33.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb32409e9f0 con 0x7fb324135950 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 10 client.134280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 10 monclient: _renew_subs 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb3240737d0 con 0x7fb324135950 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 1 librados: init done 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb3180047f0 con 0x7fb324135950 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb318004b40 con 0x7fb324135950 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb3227fc640 4 set_mon_vals no callback set 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: dump: 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.389 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb3040065d0 con 0x7fb304002870 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fb318010bb0 con 0x7fb324135950 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 client.134280.objecter ms_handle_connect 0x7fb304002870 2024-09-16T03:16:33.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 client.134280.objecter resend_mon_ops 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(459..459 src has 99..459) ==== 5367+0+0 (unknown 1657194677 0 0) 0x7fb31804b840 con 0x7fb324135950 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 client.134280.objecter ms_dispatch 0x7fb324000ba0 osd_map(459..459 src has 99..459) 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 3 client.134280.objecter handle_osd_map got epochs [459,459] > 0 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 3 client.134280.objecter handle_osd_map decoding full epoch 459 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 20 client.134280.objecter dump_active .. 0 homeless 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 737426056 0 0) 0x7fb3040065d0 con 0x7fb304002870 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient(hunting): my global_id is 134283 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb32409e750 con 0x7fb304002870 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1380047421 0 0) 0x7fb32409e750 con 0x7fb304002870 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3040065d0 con 0x7fb304002870 2024-09-16T03:16:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb314003ef0 con 0x7fb304002870 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb328e36640 10 monclient: get_auth_request con 0x7fb31003df90 auth_method 0 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: dump: 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.392 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.393 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2640683633 0 0) 0x7fb3040065d0 con 0x7fb304002870 2024-09-16T03:16:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb30400a170 con 0x7fb304002870 2024-09-16T03:16:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 client.134280.objecter ms_handle_connect 0x7fb31003df90 2024-09-16T03:16:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb314006250 con 0x7fb304002870 2024-09-16T03:16:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 >> v1:172.21.15.45:6789/0 conn(0x7fb304002870 legacy=0x7fb304004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.370+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 >> v1:172.21.15.45:6789/0 conn(0x7fb304002870 legacy=0x7fb304004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb304009e60 con 0x7fb304009120 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 client.134280.objecter ms_handle_connect 0x7fb304009120 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 client.134280.objecter resend_mon_ops 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 730723803 0 0) 0x7fb304009e60 con 0x7fb304009120 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 monclient(hunting): my global_id is 134286 2024-09-16T03:16:33.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3040065d0 con 0x7fb304009120 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2312040915 0 0) 0x7fb3040065d0 con 0x7fb304009120 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb304009e60 con 0x7fb304009120 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb3140045e0 con 0x7fb304009120 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 monclient: dump: 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.396 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.397 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.397 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.397 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3027613003 0 0) 0x7fb304009e60 con 0x7fb304009120 2024-09-16T03:16:33.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb30400c400 con 0x7fb304009120 2024-09-16T03:16:33.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4293 (unknown 0 0 3379663519) 0x7fb314005b10 con 0x7fb304009120 2024-09-16T03:16:33.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:16:33.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 >> v1:172.21.15.45:6789/0 conn(0x7fb304009120 legacy=0x7fb3040095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb320ff9640 1 -- 172.21.15.45:0/4204902765 >> v1:172.21.15.45:6789/0 conn(0x7fb304009120 legacy=0x7fb3040095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 50, 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 126606, 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:16:33.398 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:33.399 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:33.400 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:33.401 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:33.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:33.403 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:33.404 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.407 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.408 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:33.409 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.410 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.410 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:16:33.410 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 10 librados: watch_flush enter 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 10 librados: watch_flush exit 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 20 client.134280.objecter shutdown clearing up homeless session... 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 10 client.134280.objecter successfully canceled tick 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 >> v1:172.21.15.46:6812/149360280 conn(0x7fb31003df90 legacy=0x7fb310040450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 10 monclient: shutdown 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 >> v1:172.21.15.45:6789/0 conn(0x7fb324135950 legacy=0x7fb3241d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb328e36640 1 -- 172.21.15.45:0/4204902765 reap_dead start 2024-09-16T03:16:33.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb328e36640 1 -- 172.21.15.45:0/4204902765 reap_dead start 2024-09-16T03:16:33.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 shutdown_connections 2024-09-16T03:16:33.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 >> 172.21.15.45:0/4204902765 conn(0x7fb3240a2bd0 msgr2=0x7fb32413eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:33.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 shutdown_connections 2024-09-16T03:16:33.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 1 -- 172.21.15.45:0/4204902765 wait complete. 2024-09-16T03:16:33.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.378+0000 7fb32a899640 1 librados: shutdown 2024-09-16T03:16:33.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:16:33.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:16:33.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:16:33.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient: get_monmap_and_config 2024-09-16T03:16:33.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient: build_initial_monmap 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient: monmap: 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:16:33.532444+0000 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:16:33.532444+0000 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:16:33.531 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 Processor -- start 2024-09-16T03:16:33.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 -- start start 2024-09-16T03:16:33.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient: init 2024-09-16T03:16:33.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient(hunting): picked mon.noname-c con 0x7f2f20135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient(hunting): picked mon.noname-b con 0x7f2f2009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient(hunting): picked mon.noname-a con 0x7f2f2013c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2009f7e0 con 0x7f2f2013c010 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2009b160 con 0x7f2f20135950 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2009b440 con 0x7f2f2009fc40 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient(hunting): authenticate will time out at 1684.181768s 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f24a03640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f2f20135950 legacy=0x7f2f20135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f25204640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2f2013c010 0x7f2f2013e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45676/0 (socket says 172.21.15.45:45676) 2024-09-16T03:16:33.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f25204640 1 -- 172.21.15.45:0/918195063 learned_addr learned my addr 172.21.15.45:0/918195063 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 682706015 0 0) 0x7f2f2009f7e0 con 0x7f2f2013c010 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient(hunting): my global_id is 134289 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f0c003530 con 0x7f2f2013c010 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1889154537 0 0) 0x7f2f2009b440 con 0x7f2f2009fc40 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient(hunting): my global_id is 144182 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2009f7e0 con 0x7f2f2009fc40 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3481237513 0 0) 0x7f2f0c003530 con 0x7f2f2013c010 2024-09-16T03:16:33.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 >> v1:172.21.15.45:6790/0 conn(0x7f2f2009fc40 legacy=0x7f2f2013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 >> v1:172.21.15.46:6789/0 conn(0x7f2f20135950 legacy=0x7f2f20135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 monclient: found mon.noname-a 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f20136440 con 0x7f2f2013c010 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f100016a0 con 0x7f2f2013c010 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: dump: 2024-09-16T03:16:33.541 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 5 monclient: authenticate success, global_id 134289 2024-09-16T03:16:33.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f10001840 con 0x7f2f2013c010 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient: get_monmap_and_config success 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 1 -- 172.21.15.45:0/918195063 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f10001cd0 con 0x7f2f2013c010 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 4 set_mon_vals no callback set 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f1f7fe640 10 monclient: dump: 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 10 monclient: shutdown 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 -- 172.21.15.45:0/918195063 >> v1:172.21.15.45:6789/0 conn(0x7f2f2013c010 legacy=0x7f2f2013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 -- 172.21.15.45:0/918195063 shutdown_connections 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 -- 172.21.15.45:0/918195063 >> 172.21.15.45:0/918195063 conn(0x7f2f200a2bd0 msgr2=0x7f2f200a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 -- 172.21.15.45:0/918195063 shutdown_connections 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.530+0000 7f2f26c67640 1 -- 172.21.15.45:0/918195063 wait complete. 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient: build_initial_monmap 2024-09-16T03:16:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient: monmap: 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 librados: starting msgr at 2024-09-16T03:16:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 librados: starting objecter 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 Processor -- start 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 -- start start 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 librados: setting wanted keys 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 librados: calling monclient init 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient: init 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient: _reopen_session rank -1 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient(hunting): picked mon.noname-c con 0x7f2f2009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient(hunting): picked mon.noname-a con 0x7f2f20135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient(hunting): picked mon.noname-b con 0x7f2f2013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2009ee10 con 0x7f2f20135950 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f200735e0 con 0x7f2f2013c010 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2013f8f0 con 0x7f2f2009fc40 2024-09-16T03:16:33.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient(hunting): _renew_subs 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient(hunting): authenticate will time out at 1684.185032s 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f24a03640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2f2009fc40 0x7f2f200727c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43508/0 (socket says 172.21.15.45:43508) 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f24a03640 1 -- 172.21.15.45:0/1321896314 learned_addr learned my addr 172.21.15.45:0/1321896314 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f25204640 1 -- 172.21.15.45:0/1321896314 >> v1:172.21.15.46:6789/0 conn(0x7f2f2013c010 legacy=0x7f2f201d4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 client.?.objecter ms_handle_connect 0x7f2f20135950 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 client.?.objecter ms_handle_connect 0x7f2f2009fc40 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2833058506 0 0) 0x7f2f2009ee10 con 0x7f2f20135950 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): my global_id is 134292 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f00004ec0 con 0x7f2f20135950 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1800259720 0 0) 0x7f2f2013f8f0 con 0x7f2f2009fc40 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): my global_id is 144185 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2009ee10 con 0x7f2f2009fc40 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3566119096 0 0) 0x7f2f00004ec0 con 0x7f2f20135950 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f2013f8f0 con 0x7f2f20135950 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f08001860 con 0x7f2f20135950 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): dump: 2024-09-16T03:16:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3635610902 0 0) 0x7f2f2009ee10 con 0x7f2f2009fc40 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f00004ec0 con 0x7f2f2009fc40 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f18004520 con 0x7f2f2009fc40 2024-09-16T03:16:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): dump: 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2217271537 0 0) 0x7f2f2013f8f0 con 0x7f2f20135950 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 >> v1:172.21.15.45:6790/0 conn(0x7f2f2009fc40 legacy=0x7f2f200727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 >> v1:172.21.15.46:6789/0 conn(0x7f2f2013c010 legacy=0x7f2f201d4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: _finish_hunting 0 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 monclient: found mon.a 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2009d2c0 con 0x7f2f20135950 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 5 monclient: authenticate success, global_id 134292 2024-09-16T03:16:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient: _renew_subs 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f200739e0 con 0x7f2f20135950 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 client.134292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient: _renew_subs 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f20134fb0 con 0x7f2f20135950 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 librados: init done 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f08003600 con 0x7f2f20135950 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f08003950 con 0x7f2f20135950 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:16:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1effd640 4 set_mon_vals no callback set 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: dump: 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2f040065d0 con 0x7f2f04002870 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 client.134292.objecter ms_handle_connect 0x7f2f04002870 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 client.134292.objecter resend_mon_ops 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2879818559 0 0) 0x7f2f040065d0 con 0x7f2f04002870 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient(hunting): my global_id is 144188 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2013f8f0 con 0x7f2f04002870 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f2f0800f9c0 con 0x7f2f20135950 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(459..459 src has 99..459) ==== 5367+0+0 (unknown 1657194677 0 0) 0x7f2f0804a5a0 con 0x7f2f20135950 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 client.134292.objecter ms_dispatch 0x7f2f20000ba0 osd_map(459..459 src has 99..459) 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 3 client.134292.objecter handle_osd_map got epochs [459,459] > 0 2024-09-16T03:16:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 3 client.134292.objecter handle_osd_map decoding full epoch 459 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 20 client.134292.objecter dump_active .. 0 homeless 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4259113348 0 0) 0x7f2f2013f8f0 con 0x7f2f04002870 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f040065d0 con 0x7f2f04002870 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f18004e50 con 0x7f2f04002870 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 10 monclient: dump: 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1400207778 0 0) 0x7f2f040065d0 con 0x7f2f04002870 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f0400a170 con 0x7f2f04002870 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.534+0000 7f2f25204640 10 monclient: get_auth_request con 0x7f2f0003e0b0 auth_method 0 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.538+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2f180063b0 con 0x7f2f04002870 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.538+0000 7f2f1d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.538+0000 7f2f1d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.538+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 >> v1:172.21.15.45:6790/0 conn(0x7f2f04002870 legacy=0x7f2f04004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.538+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 >> v1:172.21.15.45:6790/0 conn(0x7f2f04002870 legacy=0x7f2f04004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.538+0000 7f2f1d7fa640 10 client.134292.objecter ms_handle_connect 0x7f2f0003e0b0 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2f04009e80 con 0x7f2f04009140 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 client.134292.objecter ms_handle_connect 0x7f2f04009140 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 client.134292.objecter resend_mon_ops 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 688291263 0 0) 0x7f2f04009e80 con 0x7f2f04009140 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 monclient(hunting): my global_id is 144191 2024-09-16T03:16:33.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f040065d0 con 0x7f2f04009140 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2036796451 0 0) 0x7f2f040065d0 con 0x7f2f04009140 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f04009e80 con 0x7f2f04009140 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2f180051d0 con 0x7f2f04009140 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 monclient: dump: 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:16:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2319952455 0 0) 0x7f2f04009e80 con 0x7f2f04009140 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2f0400c420 con 0x7f2f04009140 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4123 (unknown 0 0 1647795210) 0x7f2f18006560 con 0x7f2f04009140 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 >> v1:172.21.15.45:6790/0 conn(0x7f2f04009140 legacy=0x7f2f040095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f1d7fa640 1 -- 172.21.15.45:0/1321896314 >> v1:172.21.15.45:6790/0 conn(0x7f2f04009140 legacy=0x7f2f040095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 50, 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 18615, 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:33.563 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:16:33.564 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:16:33.565 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:16:33.566 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:33.567 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.568 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:16:33.569 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:16:33.570 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.570 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.573 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:16:33.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:16:33.575 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:16:33.575 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:16:33.575 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:16:33.575 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:16:33.575 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:16:33.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:16:33.575 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:16:33.575 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:16:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 10 librados: watch_flush enter 2024-09-16T03:16:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 10 librados: watch_flush exit 2024-09-16T03:16:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 20 client.134292.objecter shutdown clearing up homeless session... 2024-09-16T03:16:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 10 client.134292.objecter successfully canceled tick 2024-09-16T03:16:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 >> v1:172.21.15.46:6812/149360280 conn(0x7f2f0003e0b0 legacy=0x7f2f00040570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 10 monclient: shutdown 2024-09-16T03:16:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:16:33.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 >> v1:172.21.15.45:6789/0 conn(0x7f2f20135950 legacy=0x7f2f20072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:16:33.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f25204640 1 -- 172.21.15.45:0/1321896314 reap_dead start 2024-09-16T03:16:33.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f25204640 1 -- 172.21.15.45:0/1321896314 reap_dead start 2024-09-16T03:16:33.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 shutdown_connections 2024-09-16T03:16:33.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 >> 172.21.15.45:0/1321896314 conn(0x7f2f200a2bd0 msgr2=0x7f2f2013eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:16:33.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.542+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 shutdown_connections 2024-09-16T03:16:33.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.546+0000 7f2f26c67640 1 -- 172.21.15.45:0/1321896314 wait complete. 2024-09-16T03:16:33.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:16:33.546+0000 7f2f26c67640 1 librados: shutdown 2024-09-16T03:16:33.578 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:16:33.578 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:18:03.579 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:18:03.580 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:18:03.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:18:03.584 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:18:03.584 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:18:03.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:18:03.687 INFO:tasks.ceph.mon.b.smithi046.stderr:2024-09-16T03:18:03.683+0000 7ff3599e6e00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 495 -> 0 (sync_force? mon rebuild?) 2024-09-16T03:18:03.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient: get_monmap_and_config 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient: build_initial_monmap 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient: monmap: 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:03.721253+0000 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:03.721253+0000 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:03.720 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:03.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 1 Processor -- start 2024-09-16T03:18:03.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 1 -- start start 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient: init 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient(hunting): picked mon.noname-a con 0x7f0c040a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient(hunting): picked mon.noname-c con 0x7f0c040a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient(hunting): picked mon.noname-b con 0x7f0c04073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0409d9e0 con 0x7f0c040a4d20 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0409dc20 con 0x7f0c040a5810 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c04073360 con 0x7f0c04073d40 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient(hunting): authenticate will time out at 1774.370604s 2024-09-16T03:18:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c08ea7640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0c040a4d20 0x7f0c040a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44096/0 (socket says 172.21.15.45:44096) 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c08ea7640 1 -- 172.21.15.45:0/2288091476 learned_addr learned my addr 172.21.15.45:0/2288091476 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 -- 172.21.15.45:0/2288091476 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 78003828 0 0) 0x7f0c0409d9e0 con 0x7f0c040a4d20 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient(hunting): my global_id is 134319 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 -- 172.21.15.45:0/2288091476 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bf0003530 con 0x7f0c040a4d20 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 -- 172.21.15.45:0/2288091476 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3365432217 0 0) 0x7f0bf0003530 con 0x7f0c040a4d20 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 -- 172.21.15.45:0/2288091476 >> v1:172.21.15.45:6790/0 conn(0x7f0c04073d40 legacy=0x7f0c04140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 -- 172.21.15.45:0/2288091476 >> v1:172.21.15.46:6789/0 conn(0x7f0c040a5810 legacy=0x7f0c0413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: _finish_hunting 0 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 monclient: found mon.noname-a 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 -- 172.21.15.45:0/2288091476 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0409cf30 con 0x7f0c040a4d20 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: _finish_auth 0 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: _check_auth_tickets 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 -- 172.21.15.45:0/2288091476 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0bf4001a20 con 0x7f0c040a4d20 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 5 monclient: authenticate success, global_id 134319 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: dump: 2024-09-16T03:18:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 -- 172.21.15.45:0/2288091476 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bf4001bc0 con 0x7f0c040a4d20 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 1 -- 172.21.15.45:0/2288091476 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0bf4002010 con 0x7f0c040a4d20 2024-09-16T03:18:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c037fe640 10 monclient: dump: 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.717+0000 7f0c0b10b640 4 set_mon_vals no callback set 2024-09-16T03:18:03.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient: shutdown 2024-09-16T03:18:03.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:03.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2288091476 >> v1:172.21.15.45:6789/0 conn(0x7f0c040a4d20 legacy=0x7f0c040a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:03.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2288091476 shutdown_connections 2024-09-16T03:18:03.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2288091476 >> 172.21.15.45:0/2288091476 conn(0x7f0c040aa720 msgr2=0x7f0c040aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:03.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2288091476 shutdown_connections 2024-09-16T03:18:03.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2288091476 wait complete. 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient: build_initial_monmap 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient: monmap: 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:03.732 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 librados: starting msgr at 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 librados: starting objecter 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 Processor -- start 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 -- start start 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 librados: setting wanted keys 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 librados: calling monclient init 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient: init 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient(hunting): picked mon.noname-a con 0x7f0c04073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient(hunting): picked mon.noname-c con 0x7f0c040a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient(hunting): picked mon.noname-b con 0x7f0c040a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c04140370 con 0x7f0c04073d40 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c04140650 con 0x7f0c040a5810 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c040a3bf0 con 0x7f0c040a4d20 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c0b10b640 10 monclient(hunting): authenticate will time out at 1774.374962s 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c03fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0c040a4d20 0x7f0c041d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39004/0 (socket says 172.21.15.45:39004) 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c03fff640 1 -- 172.21.15.45:0/2960598090 learned_addr learned my addr 172.21.15.45:0/2960598090 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 client.?.objecter ms_handle_connect 0x7f0c04073d40 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 client.?.objecter ms_handle_connect 0x7f0c040a4d20 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 client.?.objecter ms_handle_connect 0x7f0c040a5810 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088787926 0 0) 0x7f0c04140370 con 0x7f0c04073d40 2024-09-16T03:18:03.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): _negotiate 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): my global_id is 134322 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0be4004ef0 con 0x7f0c04073d40 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446659822 0 0) 0x7f0c040a3bf0 con 0x7f0c040a4d20 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): _negotiate 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): my global_id is 144218 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c04140370 con 0x7f0c040a4d20 2024-09-16T03:18:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 475051200 0 0) 0x7f0c04140370 con 0x7f0c040a4d20 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c040a3bf0 con 0x7f0c040a4d20 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0bec0018c0 con 0x7f0c040a4d20 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): dump: 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:03.736 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3832118797 0 0) 0x7f0be4004ef0 con 0x7f0c04073d40 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c04140370 con 0x7f0c04073d40 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0bf4001f10 con 0x7f0c04073d40 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.721+0000 7f0c017fa640 10 monclient(hunting): dump: 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.737 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2451911782 0 0) 0x7f0c040a3bf0 con 0x7f0c040a4d20 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 >> v1:172.21.15.46:6789/0 conn(0x7f0c040a5810 legacy=0x7f0c041d5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 >> v1:172.21.15.45:6789/0 conn(0x7f0c04073d40 legacy=0x7f0c041cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 monclient: _finish_hunting 0 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 1 monclient: found mon.c 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c040a1e80 con 0x7f0c040a4d20 2024-09-16T03:18:03.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 monclient: _finish_auth 0 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 monclient: _check_auth_tickets 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 5 monclient: authenticate success, global_id 144218 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 10 monclient: _renew_subs 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c04140090 con 0x7f0c040a4d20 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 10 client.144218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 10 monclient: _renew_subs 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c0413db20 con 0x7f0c040a4d20 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 1 librados: init done 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bec003620 con 0x7f0c040a4d20 2024-09-16T03:18:03.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0bec003a30 con 0x7f0c040a4d20 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c02ffd640 4 set_mon_vals no callback set 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 monclient: dump: 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:03.740 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be8005090 con 0x7f0c040a4d20 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f0bec003bd0 con 0x7f0c040a4d20 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(535..535 src has 99..535) ==== 5869+0+0 (unknown 2956884259 0 0) 0x7f0bec04b870 con 0x7f0c040a4d20 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 client.144218.objecter ms_dispatch 0x7f0c04000ba0 osd_map(535..535 src has 99..535) 2024-09-16T03:18:03.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 3 client.144218.objecter handle_osd_map got epochs [535,535] > 0 2024-09-16T03:18:03.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 3 client.144218.objecter handle_osd_map decoding full epoch 535 2024-09-16T03:18:03.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 20 client.144218.objecter dump_active .. 0 homeless 2024-09-16T03:18:03.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c08ea7640 10 monclient: get_auth_request con 0x7f0be403e150 auth_method 0 2024-09-16T03:18:03.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.725+0000 7f0c017fa640 10 client.144218.objecter ms_handle_connect 0x7f0be403e150 2024-09-16T03:18:03.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.729+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f0bec0169d0 con 0x7f0c040a4d20 2024-09-16T03:18:03.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.729+0000 7f0c017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:03.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.729+0000 7f0c017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:03.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.977+0000 7f0c0b10b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:18:03.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.977+0000 7f0c0b10b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:18:03.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.977+0000 7f0c0b10b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:03.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.977+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0be8005380 con 0x7f0c040a4d20 2024-09-16T03:18:03.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.977+0000 7f0c017fa640 1 -- 172.21.15.45:0/2960598090 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2144131231) 0x7f0bec0162d0 con 0x7f0c040a4d20 2024-09-16T03:18:03.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.977+0000 7f0c017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:18:03.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.977+0000 7f0c017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:03.979 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:18:03.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 10 librados: watch_flush enter 2024-09-16T03:18:03.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 10 librados: watch_flush exit 2024-09-16T03:18:03.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 20 client.144218.objecter shutdown clearing up homeless session... 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 10 client.144218.objecter successfully canceled tick 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 >> v1:172.21.15.46:6812/149360280 conn(0x7f0be403e150 legacy=0x7f0be4040610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 10 monclient: shutdown 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 >> v1:172.21.15.45:6790/0 conn(0x7f0c040a4d20 legacy=0x7f0c041d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 shutdown_connections 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 >> 172.21.15.45:0/2960598090 conn(0x7f0c040aa720 msgr2=0x7f0c040a74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 shutdown_connections 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 1 -- 172.21.15.45:0/2960598090 wait complete. 2024-09-16T03:18:03.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:03.981+0000 7f0c0b10b640 1 librados: shutdown 2024-09-16T03:18:03.998 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:18:03.998 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:18:03.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:18:04.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient: get_monmap_and_config 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient: build_initial_monmap 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient: monmap: 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:04.114245+0000 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:04.114245+0000 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:04.112 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 1 Processor -- start 2024-09-16T03:18:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 1 -- start start 2024-09-16T03:18:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient: init 2024-09-16T03:18:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:18:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient(hunting): picked mon.noname-b con 0x7f74a4137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient(hunting): picked mon.noname-a con 0x7f74a4138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient(hunting): picked mon.noname-c con 0x7f74a413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a40a4f90 con 0x7f74a4138100 2024-09-16T03:18:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a40a4cb0 con 0x7f74a413e2a0 2024-09-16T03:18:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a409dc70 con 0x7f74a4137d20 2024-09-16T03:18:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a37fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f74a4137d20 0x7f74a40a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39010/0 (socket says 172.21.15.45:39010) 2024-09-16T03:18:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a2ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f74a4138100 0x7f74a413cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44110/0 (socket says 172.21.15.45:44110) 2024-09-16T03:18:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a37fe640 1 -- 172.21.15.45:0/3756344094 learned_addr learned my addr 172.21.15.45:0/3756344094 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a9cc3640 10 monclient(hunting): authenticate will time out at 1774.763577s 2024-09-16T03:18:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3190821993 0 0) 0x7f74a40a4f90 con 0x7f74a4138100 2024-09-16T03:18:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 10 monclient(hunting): my global_id is 134325 2024-09-16T03:18:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f748c003530 con 0x7f74a4138100 2024-09-16T03:18:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797747246 0 0) 0x7f74a409dc70 con 0x7f74a4137d20 2024-09-16T03:18:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 10 monclient(hunting): my global_id is 144221 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.109+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74a40a4f90 con 0x7f74a4137d20 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 380316429 0 0) 0x7f74a40a4f90 con 0x7f74a4137d20 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 >> v1:172.21.15.46:6789/0 conn(0x7f74a413e2a0 legacy=0x7f74a4140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 >> v1:172.21.15.45:6789/0 conn(0x7f74a4138100 legacy=0x7f74a413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: _finish_hunting 0 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 1 monclient: found mon.noname-b 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a409cc50 con 0x7f74a4137d20 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: _finish_auth 0 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7494001680 con 0x7f74a4137d20 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 5 monclient: authenticate success, global_id 144221 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: dump: 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.124 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74940018a0 con 0x7f74a4137d20 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 1 -- 172.21.15.45:0/3756344094 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7494001cb0 con 0x7f74a4137d20 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 4 set_mon_vals no callback set 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a27fc640 10 monclient: dump: 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient: shutdown 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3756344094 >> v1:172.21.15.45:6790/0 conn(0x7f74a4137d20 legacy=0x7f74a40a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3756344094 shutdown_connections 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3756344094 >> 172.21.15.45:0/3756344094 conn(0x7f74a40aa110 msgr2=0x7f74a40aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3756344094 shutdown_connections 2024-09-16T03:18:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3756344094 wait complete. 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient: build_initial_monmap 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient: monmap: 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 librados: starting msgr at 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 librados: starting objecter 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 Processor -- start 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 -- start start 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 librados: setting wanted keys 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 librados: calling monclient init 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient: init 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient(hunting): picked mon.noname-c con 0x7f74a4137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient(hunting): picked mon.noname-b con 0x7f74a4138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient(hunting): picked mon.noname-a con 0x7f74a413e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a4141ac0 con 0x7f74a413e2a0 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a4139740 con 0x7f74a4138100 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a40a1300 con 0x7f74a4137d20 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a9cc3640 10 monclient(hunting): authenticate will time out at 1774.766459s 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a37fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f74a4137d20 0x7f74a4138920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39014/0 (socket says 172.21.15.45:39014) 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a3fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f74a413e2a0 0x7f74a41d52b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44122/0 (socket says 172.21.15.45:44122) 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f74a37fe640 1 -- 172.21.15.45:0/3959542523 learned_addr learned my addr 172.21.15.45:0/3959542523 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 client.?.objecter ms_handle_connect 0x7f74a4137d20 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 client.?.objecter ms_handle_connect 0x7f74a413e2a0 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 client.?.objecter ms_handle_connect 0x7f74a4138100 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2689485414 0 0) 0x7f74a40a4cb0 con 0x7f74a413e2a0 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): my global_id is 134328 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7490004ec0 con 0x7f74a413e2a0 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 486110958 0 0) 0x7f7490004ec0 con 0x7f74a413e2a0 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74a40a4cb0 con 0x7f74a413e2a0 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f749c001c80 con 0x7f74a413e2a0 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): dump: 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223311058 0 0) 0x7f74a40a1300 con 0x7f74a4137d20 2024-09-16T03:18:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 10 monclient(hunting): my global_id is 144224 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.113+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7490004ec0 con 0x7f74a4137d20 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1432798735 0 0) 0x7f74a40a4cb0 con 0x7f74a413e2a0 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 >> v1:172.21.15.45:6790/0 conn(0x7f74a4137d20 legacy=0x7f74a4138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 >> v1:172.21.15.46:6789/0 conn(0x7f74a4138100 legacy=0x7f74a4139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 monclient: found mon.a 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a413a1b0 con 0x7f74a413e2a0 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: _finish_auth 0 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 5 monclient: authenticate success, global_id 134328 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 10 monclient: _renew_subs 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f74a413c510 con 0x7f74a413e2a0 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 10 client.134328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 10 monclient: _renew_subs 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f74a413cc90 con 0x7f74a413e2a0 2024-09-16T03:18:04.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 1 librados: init done 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f749c0039a0 con 0x7f74a413e2a0 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f749c003d70 con 0x7f74a413e2a0 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a1ffb640 4 set_mon_vals no callback set 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: dump: 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f74840065d0 con 0x7f7484002870 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f749c003f90 con 0x7f74a413e2a0 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 client.134328.objecter ms_handle_connect 0x7f7484002870 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 client.134328.objecter resend_mon_ops 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(536..536 src has 99..536) ==== 5861+0+0 (unknown 3801781871 0 0) 0x7f749c04be50 con 0x7f74a413e2a0 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 client.134328.objecter ms_dispatch 0x7f74a4000ba0 osd_map(536..536 src has 99..536) 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 3 client.134328.objecter handle_osd_map got epochs [536,536] > 0 2024-09-16T03:18:04.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 3 client.134328.objecter handle_osd_map decoding full epoch 536 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 20 client.134328.objecter dump_active .. 0 homeless 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876690700 0 0) 0x7f74840065d0 con 0x7f7484002870 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient(hunting): my global_id is 134331 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74a40a4cb0 con 0x7f7484002870 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 500776840 0 0) 0x7f74a40a4cb0 con 0x7f7484002870 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74840065d0 con 0x7f7484002870 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f74a2ffd640 10 monclient: get_auth_request con 0x7f749003e0b0 auth_method 0 2024-09-16T03:18:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7494001930 con 0x7f7484002870 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: dump: 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.137 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3927373400 0 0) 0x7f74840065d0 con 0x7f7484002870 2024-09-16T03:18:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f748400a170 con 0x7f7484002870 2024-09-16T03:18:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 client.134328.objecter ms_handle_connect 0x7f749003e0b0 2024-09-16T03:18:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7494001b50 con 0x7f7484002870 2024-09-16T03:18:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 >> v1:172.21.15.45:6789/0 conn(0x7f7484002870 legacy=0x7f7484004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.117+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 >> v1:172.21.15.45:6789/0 conn(0x7f7484002870 legacy=0x7f7484004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.121+0000 7f74a9cc3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:18:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7484009e60 con 0x7f7484009120 2024-09-16T03:18:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 client.134328.objecter ms_handle_connect 0x7f7484009120 2024-09-16T03:18:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 client.134328.objecter resend_mon_ops 2024-09-16T03:18:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695563808 0 0) 0x7f7484009e60 con 0x7f7484009120 2024-09-16T03:18:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 monclient(hunting): my global_id is 134334 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74840065d0 con 0x7f7484009120 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 276373346 0 0) 0x7f74840065d0 con 0x7f7484009120 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7484009e60 con 0x7f7484009120 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7494004af0 con 0x7f7484009120 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 monclient: dump: 2024-09-16T03:18:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2331091004 0 0) 0x7f7484009e60 con 0x7f7484009120 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f748400c400 con 0x7f7484009120 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4304 (unknown 0 0 2913114020) 0x7f7494004a80 con 0x7f7484009120 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:18:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 >> v1:172.21.15.45:6789/0 conn(0x7f7484009120 legacy=0x7f74840095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f748bfff640 1 -- 172.21.15.45:0/3959542523 >> v1:172.21.15.45:6789/0 conn(0x7f7484009120 legacy=0x7f74840095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:04.143 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:18:04.143 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:04.143 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:04.143 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:18:04.143 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 56, 2024-09-16T03:18:04.143 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 217353, 2024-09-16T03:18:04.143 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:18:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:18:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:04.149 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:04.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.151 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:18:04.152 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:04.153 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:18:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.156 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:18:04.156 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:18:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 10 librados: watch_flush enter 2024-09-16T03:18:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 10 librados: watch_flush exit 2024-09-16T03:18:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 20 client.134328.objecter shutdown clearing up homeless session... 2024-09-16T03:18:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 10 client.134328.objecter successfully canceled tick 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 >> v1:172.21.15.46:6812/149360280 conn(0x7f749003e0b0 legacy=0x7f7490040570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 10 monclient: shutdown 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 >> v1:172.21.15.45:6789/0 conn(0x7f74a413e2a0 legacy=0x7f74a41d52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a3fff640 1 -- 172.21.15.45:0/3959542523 reap_dead start 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a3fff640 1 -- 172.21.15.45:0/3959542523 reap_dead start 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 shutdown_connections 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 >> 172.21.15.45:0/3959542523 conn(0x7f74a40aa110 msgr2=0x7f74a4074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 shutdown_connections 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 1 -- 172.21.15.45:0/3959542523 wait complete. 2024-09-16T03:18:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.125+0000 7f74a9cc3640 1 librados: shutdown 2024-09-16T03:18:04.158 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:18:04.159 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:18:04.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:18:04.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient: get_monmap_and_config 2024-09-16T03:18:04.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient: build_initial_monmap 2024-09-16T03:18:04.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient: monmap: 2024-09-16T03:18:04.266 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:04.266 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.267 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:04.268804+0000 2024-09-16T03:18:04.267 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:04.268804+0000 2024-09-16T03:18:04.267 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:04.267 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.267 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:04.267 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:04.267 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:04.267 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 1 Processor -- start 2024-09-16T03:18:04.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 1 -- start start 2024-09-16T03:18:04.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient: init 2024-09-16T03:18:04.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient(hunting): picked mon.noname-c con 0x7faa8c137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c13e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8c0a4f90 con 0x7faa8c138100 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8c0a4cb0 con 0x7faa8c137d20 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8c09dc70 con 0x7faa8c13e2a0 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient(hunting): authenticate will time out at 1774.918767s 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8b7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7faa8c13e2a0 0x7faa8c140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39030/0 (socket says 172.21.15.45:39030) 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8b7fe640 1 -- 172.21.15.45:0/1071881096 learned_addr learned my addr 172.21.15.45:0/1071881096 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:04.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa827fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faa8c138100 0x7faa8c13cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44158/0 (socket says 172.21.15.45:44158) 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1733740785 0 0) 0x7faa8c09dc70 con 0x7faa8c13e2a0 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient(hunting): my global_id is 144227 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa78003530 con 0x7faa8c13e2a0 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1941501430 0 0) 0x7faa8c0a4f90 con 0x7faa8c138100 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient(hunting): my global_id is 134337 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa8c09dc70 con 0x7faa8c138100 2024-09-16T03:18:04.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1148206258 0 0) 0x7faa78003530 con 0x7faa8c13e2a0 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 >> v1:172.21.15.46:6789/0 conn(0x7faa8c137d20 legacy=0x7faa8c0a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 >> v1:172.21.15.45:6789/0 conn(0x7faa8c138100 legacy=0x7faa8c13cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 monclient: found mon.noname-b 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8c09cc50 con 0x7faa8c13e2a0 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7faa84001a20 con 0x7faa8c13e2a0 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 5 monclient: authenticate success, global_id 144227 2024-09-16T03:18:04.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: dump: 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.284 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa84001c40 con 0x7faa8c13e2a0 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa91423640 4 set_mon_vals no callback set 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 1 -- 172.21.15.45:0/1071881096 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7faa84002090 con 0x7faa8c13e2a0 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.265+0000 7faa8a7fc640 10 monclient: dump: 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: shutdown 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- 172.21.15.45:0/1071881096 >> v1:172.21.15.45:6790/0 conn(0x7faa8c13e2a0 legacy=0x7faa8c140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- 172.21.15.45:0/1071881096 shutdown_connections 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- 172.21.15.45:0/1071881096 >> 172.21.15.45:0/1071881096 conn(0x7faa8c0aa110 msgr2=0x7faa8c0aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- 172.21.15.45:0/1071881096 shutdown_connections 2024-09-16T03:18:04.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- 172.21.15.45:0/1071881096 wait complete. 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: build_initial_monmap 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: monmap: 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 librados: starting msgr at 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 librados: starting objecter 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 Processor -- start 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- start start 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 librados: setting wanted keys 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 librados: calling monclient init 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: init 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient(hunting): picked mon.noname-c con 0x7faa8c13e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8c0a1070 con 0x7faa8c137d20 2024-09-16T03:18:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8c1389c0 con 0x7faa8c138100 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8c09eea0 con 0x7faa8c13e2a0 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient(hunting): authenticate will time out at 1774.921705s 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa8affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faa8c137d20 0x7faa8c09e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44168/0 (socket says 172.21.15.45:44168) 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa8affd640 1 -- 172.21.15.45:0/257308765 learned_addr learned my addr 172.21.15.45:0/257308765 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 client.?.objecter ms_handle_connect 0x7faa8c13e2a0 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 client.?.objecter ms_handle_connect 0x7faa8c137d20 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786020042 0 0) 0x7faa8c09eea0 con 0x7faa8c13e2a0 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): my global_id is 144230 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa6c004ec0 con 0x7faa8c13e2a0 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872541581 0 0) 0x7faa8c0a1070 con 0x7faa8c137d20 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): my global_id is 134340 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa8c09eea0 con 0x7faa8c137d20 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 client.?.objecter ms_handle_connect 0x7faa8c138100 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1738974274 0 0) 0x7faa6c004ec0 con 0x7faa8c13e2a0 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa8c0a1070 con 0x7faa8c13e2a0 2024-09-16T03:18:04.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7faa84001df0 con 0x7faa8c13e2a0 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): dump: 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.291 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:04.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3912092089 0 0) 0x7faa8c09eea0 con 0x7faa8c137d20 2024-09-16T03:18:04.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa6c004ec0 con 0x7faa8c137d20 2024-09-16T03:18:04.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7faa7c0019a0 con 0x7faa8c137d20 2024-09-16T03:18:04.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:04.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:04.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): dump: 2024-09-16T03:18:04.292 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 223924997 0 0) 0x7faa8c0a1070 con 0x7faa8c13e2a0 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 >> v1:172.21.15.46:6789/0 conn(0x7faa8c138100 legacy=0x7faa8c09e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 >> v1:172.21.15.45:6789/0 conn(0x7faa8c137d20 legacy=0x7faa8c09e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: _finish_hunting 0 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 monclient: found mon.c 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8c1376f0 con 0x7faa8c13e2a0 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: _finish_auth 0 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: _check_auth_tickets 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 5 monclient: authenticate success, global_id 144230 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: _renew_subs 2024-09-16T03:18:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa8c0a1300 con 0x7faa8c13e2a0 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 client.144230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: _renew_subs 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7faa8c131320 con 0x7faa8c13e2a0 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 librados: init done 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa84001ba0 con 0x7faa8c13e2a0 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7faa84005000 con 0x7faa8c13e2a0 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa897fa640 4 set_mon_vals no callback set 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: dump: 2024-09-16T03:18:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faa700065d0 con 0x7faa70002870 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 client.144230.objecter ms_handle_connect 0x7faa70002870 2024-09-16T03:18:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 client.144230.objecter resend_mon_ops 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864144841 0 0) 0x7faa700065d0 con 0x7faa70002870 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient(hunting): my global_id is 144233 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa8c0a1070 con 0x7faa70002870 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7faa84011070 con 0x7faa8c13e2a0 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(536..536 src has 99..536) ==== 5861+0+0 (unknown 3801781871 0 0) 0x7faa8404bc40 con 0x7faa8c13e2a0 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 client.144230.objecter ms_dispatch 0x7faa8c000ba0 osd_map(536..536 src has 99..536) 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 3 client.144230.objecter handle_osd_map got epochs [536,536] > 0 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 3 client.144230.objecter handle_osd_map decoding full epoch 536 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 20 client.144230.objecter dump_active .. 0 homeless 2024-09-16T03:18:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3660475455 0 0) 0x7faa8c0a1070 con 0x7faa70002870 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa700065d0 con 0x7faa70002870 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7faa7c0037c0 con 0x7faa70002870 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.269+0000 7faa83fff640 10 monclient: dump: 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.273+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 351348578 0 0) 0x7faa700065d0 con 0x7faa70002870 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.273+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa7000a170 con 0x7faa70002870 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.273+0000 7faa827fc640 10 monclient: get_auth_request con 0x7faa6c03e140 auth_method 0 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.273+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7faa7c0057d0 con 0x7faa70002870 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.273+0000 7faa83fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.273+0000 7faa83fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.273+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 >> v1:172.21.15.45:6790/0 conn(0x7faa70002870 legacy=0x7faa70004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.273+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 >> v1:172.21.15.45:6790/0 conn(0x7faa70002870 legacy=0x7faa70004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.273+0000 7faa83fff640 10 client.144230.objecter ms_handle_connect 0x7faa6c03e140 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faa70009e60 con 0x7faa70009120 2024-09-16T03:18:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 client.144230.objecter ms_handle_connect 0x7faa70009120 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 client.144230.objecter resend_mon_ops 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 310214493 0 0) 0x7faa70009e60 con 0x7faa70009120 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 monclient(hunting): my global_id is 144236 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa700065d0 con 0x7faa70009120 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1285820128 0 0) 0x7faa700065d0 con 0x7faa70009120 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa70009e60 con 0x7faa70009120 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7faa7c003d30 con 0x7faa70009120 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 monclient: dump: 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1178380514 0 0) 0x7faa70009e60 con 0x7faa70009120 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa7000c3b0 con 0x7faa70009120 2024-09-16T03:18:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 2382255228) 0x7faa7c003d60 con 0x7faa70009120 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 >> v1:172.21.15.45:6790/0 conn(0x7faa70009120 legacy=0x7faa700095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa83fff640 1 -- 172.21.15.45:0/257308765 >> v1:172.21.15.45:6790/0 conn(0x7faa70009120 legacy=0x7faa700095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 56, 2024-09-16T03:18:04.302 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 109350, 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:04.303 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:18:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:04.306 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:18:04.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:04.309 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:04.312 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:04.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.312 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:04.312 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:18:04.313 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:04.313 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.313 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.313 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:18:04.313 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:18:04.313 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:04.313 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:04.314 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.316 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.316 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:18:04.316 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:18:04.316 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:04.316 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:04.316 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.317 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:18:04.318 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 10 librados: watch_flush enter 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 10 librados: watch_flush exit 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 20 client.144230.objecter shutdown clearing up homeless session... 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 10 client.144230.objecter successfully canceled tick 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 >> v1:172.21.15.46:6812/149360280 conn(0x7faa6c03e140 legacy=0x7faa6c040600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 10 monclient: shutdown 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 >> v1:172.21.15.45:6790/0 conn(0x7faa8c13e2a0 legacy=0x7faa8c1d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa8b7fe640 1 -- 172.21.15.45:0/257308765 reap_dead start 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa8b7fe640 1 -- 172.21.15.45:0/257308765 reap_dead start 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 shutdown_connections 2024-09-16T03:18:04.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 >> 172.21.15.45:0/257308765 conn(0x7faa8c0aa110 msgr2=0x7faa8c0731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:04.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 shutdown_connections 2024-09-16T03:18:04.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 1 -- 172.21.15.45:0/257308765 wait complete. 2024-09-16T03:18:04.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.277+0000 7faa91423640 1 librados: shutdown 2024-09-16T03:18:04.322 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:18:04.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:18:04.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:18:04.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.433+0000 7f6107f1b640 10 monclient: get_monmap_and_config 2024-09-16T03:18:04.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.433+0000 7f6107f1b640 10 monclient: build_initial_monmap 2024-09-16T03:18:04.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.433+0000 7f6107f1b640 10 monclient: monmap: 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:04.439870+0000 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:04.439870+0000 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:04.438 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 Processor -- start 2024-09-16T03:18:04.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 -- start start 2024-09-16T03:18:04.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient: init 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient(hunting): picked mon.noname-a con 0x7f6100137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient(hunting): picked mon.noname-b con 0x7f6100138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient(hunting): picked mon.noname-c con 0x7f610013e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61000a4f90 con 0x7f6100137d20 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61000a4cb0 con 0x7f610013e2a0 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610009dc70 con 0x7f6100138100 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient(hunting): authenticate will time out at 1775.089180s 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f61054b6640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6100138100 0x7f610013cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39072/0 (socket says 172.21.15.45:39072) 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f61054b6640 1 -- 172.21.15.45:0/2806482592 learned_addr learned my addr 172.21.15.45:0/2806482592 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:04.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6105cb7640 1 --1- 172.21.15.45:0/2806482592 >> v1:172.21.15.45:6789/0 conn(0x7f6100137d20 0x7f61000a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44178/0 (socket says 172.21.15.45:44178) 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 166941118 0 0) 0x7f610009dc70 con 0x7f6100138100 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient(hunting): my global_id is 144239 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60ec003530 con 0x7f6100138100 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 254868820 0 0) 0x7f61000a4f90 con 0x7f6100137d20 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient(hunting): my global_id is 134343 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f610009dc70 con 0x7f6100137d20 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 710133046 0 0) 0x7f610009dc70 con 0x7f6100137d20 2024-09-16T03:18:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 >> v1:172.21.15.45:6790/0 conn(0x7f6100138100 legacy=0x7f610013cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 >> v1:172.21.15.46:6789/0 conn(0x7f610013e2a0 legacy=0x7f6100140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: _finish_hunting 0 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 monclient: found mon.noname-a 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610009cc50 con 0x7f6100137d20 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: _finish_auth 0 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: _check_auth_tickets 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60f0001680 con 0x7f6100137d20 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 5 monclient: authenticate success, global_id 134343 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: dump: 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60f0001820 con 0x7f6100137d20 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 1 -- 172.21.15.45:0/2806482592 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60f0001cb0 con 0x7f6100137d20 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6104cb5640 10 monclient: dump: 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 4 set_mon_vals no callback set 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient: shutdown 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 -- 172.21.15.45:0/2806482592 >> v1:172.21.15.45:6789/0 conn(0x7f6100137d20 legacy=0x7f61000a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 -- 172.21.15.45:0/2806482592 shutdown_connections 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 -- 172.21.15.45:0/2806482592 >> 172.21.15.45:0/2806482592 conn(0x7f61000aa110 msgr2=0x7f61000aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 -- 172.21.15.45:0/2806482592 shutdown_connections 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 -- 172.21.15.45:0/2806482592 wait complete. 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient: build_initial_monmap 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 10 monclient: monmap: 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 librados: starting msgr at 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.437+0000 7f6107f1b640 1 librados: starting objecter 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 Processor -- start 2024-09-16T03:18:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 -- start start 2024-09-16T03:18:04.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 librados: setting wanted keys 2024-09-16T03:18:04.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 librados: calling monclient init 2024-09-16T03:18:04.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient: init 2024-09-16T03:18:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:18:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient(hunting): picked mon.noname-a con 0x7f6100137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient(hunting): picked mon.noname-c con 0x7f6100138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient(hunting): picked mon.noname-b con 0x7f610013e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610013bbf0 con 0x7f6100137d20 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6100139770 con 0x7f610013e2a0 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61000a1300 con 0x7f6100138100 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient(hunting): authenticate will time out at 1775.092726s 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f61054b6640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6100138100 0x7f6100139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39078/0 (socket says 172.21.15.45:39078) 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6105cb7640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6100137d20 0x7f6100138950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44190/0 (socket says 172.21.15.45:44190) 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f61054b6640 1 -- 172.21.15.45:0/264239599 learned_addr learned my addr 172.21.15.45:0/264239599 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.?.objecter ms_handle_connect 0x7f6100137d20 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.?.objecter ms_handle_connect 0x7f6100138100 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 737071817 0 0) 0x7f61000a1300 con 0x7f6100138100 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): my global_id is 144242 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60e4004ec0 con 0x7f6100138100 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3308583995 0 0) 0x7f610013bbf0 con 0x7f6100137d20 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): my global_id is 134346 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61000a1300 con 0x7f6100137d20 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.?.objecter ms_handle_connect 0x7f610013e2a0 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2062103920 0 0) 0x7f61000a1300 con 0x7f6100137d20 2024-09-16T03:18:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f610013bbf0 con 0x7f6100137d20 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3682463588 0 0) 0x7f60e4004ec0 con 0x7f6100138100 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61000a1300 con 0x7f6100138100 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60f0001c30 con 0x7f6100137d20 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): dump: 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60fc0013a0 con 0x7f6100138100 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): dump: 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3698733480 0 0) 0x7f610013bbf0 con 0x7f6100137d20 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 >> v1:172.21.15.45:6790/0 conn(0x7f6100138100 legacy=0x7f6100139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 >> v1:172.21.15.46:6789/0 conn(0x7f610013e2a0 legacy=0x7f61001d4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: _finish_hunting 0 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 monclient: found mon.a 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610013b910 con 0x7f6100137d20 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: _finish_auth 0 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 5 monclient: authenticate success, global_id 134346 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient: _renew_subs 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f610013a830 con 0x7f6100137d20 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60f0004aa0 con 0x7f6100137d20 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 client.134346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60f0004eb0 con 0x7f6100137d20 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient: _renew_subs 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f610013a3f0 con 0x7f6100137d20 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 librados: init done 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f7fff640 4 set_mon_vals no callback set 2024-09-16T03:18:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: dump: 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f610013afc0 con 0x7f61000a14f0 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f60f0010fa0 con 0x7f6100137d20 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.134346.objecter ms_handle_connect 0x7f61000a14f0 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.134346.objecter resend_mon_ops 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(536..536 src has 99..536) ==== 5861+0+0 (unknown 3801781871 0 0) 0x7f60f004b800 con 0x7f6100137d20 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.134346.objecter ms_dispatch 0x7f6100000ba0 osd_map(536..536 src has 99..536) 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 3 client.134346.objecter handle_osd_map got epochs [536,536] > 0 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 3 client.134346.objecter handle_osd_map decoding full epoch 536 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 20 client.134346.objecter dump_active .. 0 homeless 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1683893652 0 0) 0x7f610013afc0 con 0x7f61000a14f0 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient(hunting): my global_id is 154111 2024-09-16T03:18:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f610013bbf0 con 0x7f61000a14f0 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f61064b8640 10 monclient: get_auth_request con 0x7f60e403e020 auth_method 0 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 client.134346.objecter ms_handle_connect 0x7f60e403e020 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2540919169 0 0) 0x7f610013bbf0 con 0x7f61000a14f0 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f610013afc0 con 0x7f61000a14f0 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60fc004c00 con 0x7f61000a14f0 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.441+0000 7f60f67fc640 10 monclient: dump: 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.445+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1252648197 0 0) 0x7f610013afc0 con 0x7f61000a14f0 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.445+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61001418d0 con 0x7f61000a14f0 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.445+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f60fc004e20 con 0x7f61000a14f0 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.445+0000 7f60f67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.445+0000 7f60f67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.445+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 >> v1:172.21.15.46:6789/0 conn(0x7f61000a14f0 legacy=0x7f61000a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.445+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 >> v1:172.21.15.46:6789/0 conn(0x7f61000a14f0 legacy=0x7f61000a18d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f6107f1b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:18:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61000767c0 con 0x7f61000a1fe0 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f60f67fc640 10 client.134346.objecter ms_handle_connect 0x7f61000a1fe0 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f60f67fc640 10 client.134346.objecter resend_mon_ops 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934125952 0 0) 0x7f61000767c0 con 0x7f61000a1fe0 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f60f67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f60f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f60f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f60f67fc640 10 monclient(hunting): my global_id is 154114 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.449+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f610013afc0 con 0x7f61000a1fe0 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1285347393 0 0) 0x7f610013afc0 con 0x7f61000a1fe0 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61000767c0 con 0x7f61000a1fe0 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f60fc0054a0 con 0x7f61000a1fe0 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 10 monclient: dump: 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1685613888 0 0) 0x7f61000767c0 con 0x7f61000a1fe0 2024-09-16T03:18:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f61000a23e0 con 0x7f61000a1fe0 2024-09-16T03:18:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3965 (unknown 0 0 4194488319) 0x7f60fc006990 con 0x7f61000a1fe0 2024-09-16T03:18:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:18:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 >> v1:172.21.15.46:6789/0 conn(0x7f61000a1fe0 legacy=0x7f61001da4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f60f67fc640 1 -- 172.21.15.45:0/264239599 >> v1:172.21.15.46:6789/0 conn(0x7f61000a1fe0 legacy=0x7f61001da4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 56, 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 783, 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.468 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:04.469 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:04.470 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.470 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:18:04.470 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:18:04.470 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:04.470 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:04.470 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:04.470 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:04.470 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:04.470 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:18:04.471 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:04.472 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.473 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:18:04.474 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.475 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:04.476 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:04.477 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:18:04.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:04.479 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:04.479 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:04.479 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:04.479 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:04.479 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:04.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:04.479 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:18:04.479 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:18:04.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 10 librados: watch_flush enter 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 10 librados: watch_flush exit 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 20 client.134346.objecter shutdown clearing up homeless session... 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 10 client.134346.objecter successfully canceled tick 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 >> v1:172.21.15.46:6812/149360280 conn(0x7f60e403e020 legacy=0x7f60e40404e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 10 monclient: shutdown 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 >> v1:172.21.15.45:6789/0 conn(0x7f6100137d20 legacy=0x7f6100138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f61064b8640 1 -- 172.21.15.45:0/264239599 reap_dead start 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f61064b8640 1 -- 172.21.15.45:0/264239599 reap_dead start 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 shutdown_connections 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 >> 172.21.15.45:0/264239599 conn(0x7f61000aa110 msgr2=0x7f61000746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 shutdown_connections 2024-09-16T03:18:04.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 1 -- 172.21.15.45:0/264239599 wait complete. 2024-09-16T03:18:04.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.453+0000 7f6107f1b640 1 librados: shutdown 2024-09-16T03:18:04.482 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:18:04.482 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:18:04.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:18:04.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.597+0000 7fca970b3640 10 monclient: get_monmap_and_config 2024-09-16T03:18:04.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.597+0000 7fca970b3640 10 monclient: build_initial_monmap 2024-09-16T03:18:04.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.597+0000 7fca970b3640 10 monclient: monmap: 2024-09-16T03:18:04.600 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:04.602892+0000 2024-09-16T03:18:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:04.602892+0000 2024-09-16T03:18:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:04.601 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 Processor -- start 2024-09-16T03:18:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- start start 2024-09-16T03:18:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: init 2024-09-16T03:18:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:18:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient(hunting): picked mon.noname-a con 0x7fca90137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient(hunting): picked mon.noname-b con 0x7fca90138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient(hunting): picked mon.noname-c con 0x7fca9013e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca900a4f90 con 0x7fca90137d20 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca900a4cb0 con 0x7fca9013e2a0 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca9009dc70 con 0x7fca90138100 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient(hunting): authenticate will time out at 1775.252848s 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca94e4f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fca90137d20 0x7fca900a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44198/0 (socket says 172.21.15.45:44198) 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8ffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fca90138100 0x7fca9013cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39084/0 (socket says 172.21.15.45:39084) 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca94e4f640 1 -- 172.21.15.45:0/3702702761 learned_addr learned my addr 172.21.15.45:0/3702702761 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378411507 0 0) 0x7fca900a4f90 con 0x7fca90137d20 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient(hunting): my global_id is 134349 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca7c003530 con 0x7fca90137d20 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714401090 0 0) 0x7fca9009dc70 con 0x7fca90138100 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient(hunting): my global_id is 144245 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca900a4f90 con 0x7fca90138100 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 897291486 0 0) 0x7fca7c003530 con 0x7fca90137d20 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 >> v1:172.21.15.45:6790/0 conn(0x7fca90138100 legacy=0x7fca9013cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 >> v1:172.21.15.46:6789/0 conn(0x7fca9013e2a0 legacy=0x7fca90140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 monclient: found mon.noname-a 2024-09-16T03:18:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9009cc50 con 0x7fca90137d20 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fca80001a20 con 0x7fca90137d20 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 5 monclient: authenticate success, global_id 134349 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: dump: 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca80001c40 con 0x7fca90137d20 2024-09-16T03:18:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 1 -- 172.21.15.45:0/3702702761 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fca800020d0 con 0x7fca90137d20 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca8f7fe640 10 monclient: dump: 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 4 set_mon_vals no callback set 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: shutdown 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- 172.21.15.45:0/3702702761 >> v1:172.21.15.45:6789/0 conn(0x7fca90137d20 legacy=0x7fca900a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- 172.21.15.45:0/3702702761 shutdown_connections 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- 172.21.15.45:0/3702702761 >> 172.21.15.45:0/3702702761 conn(0x7fca900aa110 msgr2=0x7fca900aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- 172.21.15.45:0/3702702761 shutdown_connections 2024-09-16T03:18:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- 172.21.15.45:0/3702702761 wait complete. 2024-09-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: build_initial_monmap 2024-09-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: monmap: 2024-09-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:04.612 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:04.612 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:04.612 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 librados: starting msgr at 2024-09-16T03:18:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 librados: starting objecter 2024-09-16T03:18:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 Processor -- start 2024-09-16T03:18:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- start start 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 librados: setting wanted keys 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 librados: calling monclient init 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: init 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient(hunting): picked mon.noname-a con 0x7fca90137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient(hunting): picked mon.noname-c con 0x7fca90138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 10 monclient(hunting): picked mon.noname-b con 0x7fca9013e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.601+0000 7fca970b3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca90137590 con 0x7fca90137d20 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca90141af0 con 0x7fca9013e2a0 2024-09-16T03:18:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca900a1300 con 0x7fca90138100 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca94e4f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fca90137d20 0x7fca9013d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44214/0 (socket says 172.21.15.45:44214) 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 monclient(hunting): authenticate will time out at 1775.256097s 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca94e4f640 1 -- 172.21.15.45:0/443741999 learned_addr learned my addr 172.21.15.45:0/443741999 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 client.?.objecter ms_handle_connect 0x7fca90137d20 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 client.?.objecter ms_handle_connect 0x7fca90138100 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4251650266 0 0) 0x7fca90137590 con 0x7fca90137d20 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): my global_id is 134352 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca70004ec0 con 0x7fca90137d20 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2244159611 0 0) 0x7fca900a1300 con 0x7fca90138100 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): my global_id is 144248 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca90137590 con 0x7fca90138100 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 client.?.objecter ms_handle_connect 0x7fca9013e2a0 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3664694433 0 0) 0x7fca70004ec0 con 0x7fca90137d20 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca900a1300 con 0x7fca90137d20 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fca80002050 con 0x7fca90137d20 2024-09-16T03:18:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): dump: 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 769526185 0 0) 0x7fca90137590 con 0x7fca90138100 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca70004ec0 con 0x7fca90138100 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fca78001190 con 0x7fca90138100 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): dump: 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3207879180 0 0) 0x7fca900a4cb0 con 0x7fca9013e2a0 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _negotiate 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient(hunting): my global_id is 154120 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca90137590 con 0x7fca9013e2a0 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1413905885 0 0) 0x7fca900a1300 con 0x7fca90137d20 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 >> v1:172.21.15.45:6790/0 conn(0x7fca90138100 legacy=0x7fca901d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 >> v1:172.21.15.46:6789/0 conn(0x7fca9013e2a0 legacy=0x7fca901d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient: _finish_hunting 0 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 monclient: found mon.a 2024-09-16T03:18:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9013f8d0 con 0x7fca90137d20 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 5 monclient: authenticate success, global_id 134352 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 monclient: _renew_subs 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca90138e30 con 0x7fca90137d20 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 client.134352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 monclient: _renew_subs 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fca901397e0 con 0x7fca90137d20 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 1 librados: init done 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca80001ba0 con 0x7fca90137d20 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fca80004fc0 con 0x7fca90137d20 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8effd640 4 set_mon_vals no callback set 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 monclient: dump: 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:04.624 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca74005090 con 0x7fca90137d20 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fca80005160 con 0x7fca90137d20 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(536..536 src has 99..536) ==== 5861+0+0 (unknown 3801781871 0 0) 0x7fca8004bd50 con 0x7fca90137d20 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 client.134352.objecter ms_dispatch 0x7fca90000ba0 osd_map(536..536 src has 99..536) 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 3 client.134352.objecter handle_osd_map got epochs [536,536] > 0 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 3 client.134352.objecter handle_osd_map decoding full epoch 536 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 20 client.134352.objecter dump_active .. 0 homeless 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8ffff640 10 monclient: get_auth_request con 0x7fca7003e2b0 auth_method 0 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.605+0000 7fca8d7fa640 10 client.134352.objecter ms_handle_connect 0x7fca7003e2b0 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.609+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fca90138e30 con 0x7fca90137d20 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.609+0000 7fca8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.609+0000 7fca8d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:04.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.845+0000 7fca970b3640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:18:04.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.845+0000 7fca970b3640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:18:04.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.845+0000 7fca970b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:18:04.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.845+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fca74005c50 con 0x7fca90137d20 2024-09-16T03:18:04.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.845+0000 7fca8d7fa640 1 -- 172.21.15.45:0/443741999 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fca80016f10 con 0x7fca90137d20 2024-09-16T03:18:04.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.845+0000 7fca8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:18:04.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.845+0000 7fca8d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:04.849 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.845+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.849 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.845+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=14,logm=53} crc {auth=4217993090,config=1097430493,health=4181180072,kv=3374818887,logm=3065801094}) 2024-09-16T03:18:04.849 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.845+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=1,health=12,kv=13,logm=50} crc {auth=4014368114,config=1228700967,health=4181180072,kv=3993304674,logm=2770999982}) 2024-09-16T03:18:04.850 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.845+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.850 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.845+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=14,logm=53} crc {auth=4217993090,config=1097430493,health=4181180072,kv=3374818887,logm=3065801094}) 2024-09-16T03:18:04.850 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.845+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=3687829281,config=1097430493,health=4181180072,kv=611822657,logm=1040588660}) 2024-09-16T03:18:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 10 librados: watch_flush enter 2024-09-16T03:18:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 10 librados: watch_flush exit 2024-09-16T03:18:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 20 client.134352.objecter shutdown clearing up homeless session... 2024-09-16T03:18:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 10 client.134352.objecter successfully canceled tick 2024-09-16T03:18:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 >> v1:172.21.15.46:6812/149360280 conn(0x7fca7003e2b0 legacy=0x7fca70040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 10 monclient: shutdown 2024-09-16T03:18:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 >> v1:172.21.15.45:6789/0 conn(0x7fca90137d20 legacy=0x7fca9013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 shutdown_connections 2024-09-16T03:18:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 >> 172.21.15.45:0/443741999 conn(0x7fca900aa110 msgr2=0x7fca900731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 shutdown_connections 2024-09-16T03:18:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 1 -- 172.21.15.45:0/443741999 wait complete. 2024-09-16T03:18:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:04.849+0000 7fca970b3640 1 librados: shutdown 2024-09-16T03:18:04.853 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.853 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998059428}) 2024-09-16T03:18:04.853 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=422609900}) 2024-09-16T03:18:04.853 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.853 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998059428}) 2024-09-16T03:18:04.854 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=992617255}) 2024-09-16T03:18:04.854 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.854 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117890437}) 2024-09-16T03:18:04.854 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2794509473}) 2024-09-16T03:18:04.854 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.854 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117890437}) 2024-09-16T03:18:04.854 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.849+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4033928264}) 2024-09-16T03:18:04.855 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.855 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781387616}) 2024-09-16T03:18:04.855 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1938286529}) 2024-09-16T03:18:04.855 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.855 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781387616}) 2024-09-16T03:18:04.855 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=570677859}) 2024-09-16T03:18:04.856 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.856 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000822235}) 2024-09-16T03:18:04.856 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=255846518}) 2024-09-16T03:18:04.856 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.856 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000822235}) 2024-09-16T03:18:04.856 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.853+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133113002}) 2024-09-16T03:18:04.857 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.858 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2664693800}) 2024-09-16T03:18:04.858 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=899704266}) 2024-09-16T03:18:04.858 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.858 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2664693800}) 2024-09-16T03:18:04.858 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3128586131}) 2024-09-16T03:18:04.859 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.859 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536713939}) 2024-09-16T03:18:04.859 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=484677273}) 2024-09-16T03:18:04.859 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.859 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536713939}) 2024-09-16T03:18:04.859 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959803197}) 2024-09-16T03:18:04.860 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.860 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964516565}) 2024-09-16T03:18:04.860 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3073025865}) 2024-09-16T03:18:04.860 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.860 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964516565}) 2024-09-16T03:18:04.861 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.857+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2321188386}) 2024-09-16T03:18:04.861 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.861 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888492855}) 2024-09-16T03:18:04.861 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1374412843}) 2024-09-16T03:18:04.862 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.862 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888492855}) 2024-09-16T03:18:04.862 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451232406}) 2024-09-16T03:18:04.863 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.863 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836875348}) 2024-09-16T03:18:04.863 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3048960017}) 2024-09-16T03:18:04.863 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.863 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836875348}) 2024-09-16T03:18:04.863 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1239149388}) 2024-09-16T03:18:04.864 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.864 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323368203}) 2024-09-16T03:18:04.864 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280588438}) 2024-09-16T03:18:04.864 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.864 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323368203}) 2024-09-16T03:18:04.865 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=301644995}) 2024-09-16T03:18:04.865 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.865 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.861+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3980987056}) 2024-09-16T03:18:04.866 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2144109034}) 2024-09-16T03:18:04.866 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.866 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3980987056}) 2024-09-16T03:18:04.866 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976599729}) 2024-09-16T03:18:04.867 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:18:04.868 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.868 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335787820}) 2024-09-16T03:18:04.868 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=784135550}) 2024-09-16T03:18:04.868 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.868 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335787820}) 2024-09-16T03:18:04.868 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4034888420}) 2024-09-16T03:18:04.869 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.869 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81607236}) 2024-09-16T03:18:04.869 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1797283519}) 2024-09-16T03:18:04.869 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.869 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81607236}) 2024-09-16T03:18:04.869 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.865+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=956633545}) 2024-09-16T03:18:04.870 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.870 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3523464427}) 2024-09-16T03:18:04.870 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955885024}) 2024-09-16T03:18:04.870 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.870 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3523464427}) 2024-09-16T03:18:04.870 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2513646563}) 2024-09-16T03:18:04.871 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.871 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244029212}) 2024-09-16T03:18:04.871 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=753625257}) 2024-09-16T03:18:04.871 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.871 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244029212}) 2024-09-16T03:18:04.871 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470472015}) 2024-09-16T03:18:04.872 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.872 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068232324}) 2024-09-16T03:18:04.872 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=617587441}) 2024-09-16T03:18:04.872 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.873 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068232324}) 2024-09-16T03:18:04.873 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.869+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2804745580}) 2024-09-16T03:18:04.873 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.874 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377007434}) 2024-09-16T03:18:04.874 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=89382191}) 2024-09-16T03:18:04.874 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.874 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377007434}) 2024-09-16T03:18:04.874 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2672219416}) 2024-09-16T03:18:04.875 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.875 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425482504}) 2024-09-16T03:18:04.875 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1250943684}) 2024-09-16T03:18:04.875 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.875 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425482504}) 2024-09-16T03:18:04.876 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567916990}) 2024-09-16T03:18:04.877 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.878 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90132841}) 2024-09-16T03:18:04.878 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4127581451}) 2024-09-16T03:18:04.878 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.878 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90132841}) 2024-09-16T03:18:04.878 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.873+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1819247023}) 2024-09-16T03:18:04.878 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.879 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293153002}) 2024-09-16T03:18:04.879 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2435371392}) 2024-09-16T03:18:04.879 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.879 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293153002}) 2024-09-16T03:18:04.879 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4159060034}) 2024-09-16T03:18:04.880 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.880 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005000385}) 2024-09-16T03:18:04.880 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3598416599}) 2024-09-16T03:18:04.880 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.880 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005000385}) 2024-09-16T03:18:04.880 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2006571833}) 2024-09-16T03:18:04.881 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.881 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662630726}) 2024-09-16T03:18:04.882 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.877+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=637834629}) 2024-09-16T03:18:04.882 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.882 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662630726}) 2024-09-16T03:18:04.882 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922431002}) 2024-09-16T03:18:04.883 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.883 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213865424}) 2024-09-16T03:18:04.883 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2239809467}) 2024-09-16T03:18:04.883 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.883 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213865424}) 2024-09-16T03:18:04.883 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512733122}) 2024-09-16T03:18:04.884 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.884 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415910741}) 2024-09-16T03:18:04.884 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1203867543}) 2024-09-16T03:18:04.884 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.884 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415910741}) 2024-09-16T03:18:04.884 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2080897989}) 2024-09-16T03:18:04.885 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.885 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1647779752}) 2024-09-16T03:18:04.885 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2191221047}) 2024-09-16T03:18:04.886 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.886 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1647779752}) 2024-09-16T03:18:04.886 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.881+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3301631614}) 2024-09-16T03:18:04.887 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.887 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626937912}) 2024-09-16T03:18:04.887 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648070142}) 2024-09-16T03:18:04.887 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.887 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626937912}) 2024-09-16T03:18:04.887 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=915045996}) 2024-09-16T03:18:04.888 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.889 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385935972}) 2024-09-16T03:18:04.889 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4076083882}) 2024-09-16T03:18:04.889 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.889 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385935972}) 2024-09-16T03:18:04.889 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.885+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330615992}) 2024-09-16T03:18:04.890 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.890 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280581509}) 2024-09-16T03:18:04.890 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3237737715}) 2024-09-16T03:18:04.890 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.890 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280581509}) 2024-09-16T03:18:04.890 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919764118}) 2024-09-16T03:18:04.891 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.891 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602581382}) 2024-09-16T03:18:04.891 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3416976883}) 2024-09-16T03:18:04.891 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.891 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602581382}) 2024-09-16T03:18:04.891 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939633440}) 2024-09-16T03:18:04.892 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.892 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807173902}) 2024-09-16T03:18:04.893 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4252870130}) 2024-09-16T03:18:04.893 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.893 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807173902}) 2024-09-16T03:18:04.893 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.889+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3825805086}) 2024-09-16T03:18:04.894 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.894 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165596375}) 2024-09-16T03:18:04.894 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2756772801}) 2024-09-16T03:18:04.894 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.894 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165596375}) 2024-09-16T03:18:04.894 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1704711077}) 2024-09-16T03:18:04.895 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.895 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076869350}) 2024-09-16T03:18:04.895 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2429320036}) 2024-09-16T03:18:04.895 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.895 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076869350}) 2024-09-16T03:18:04.896 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1271192457}) 2024-09-16T03:18:04.897 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.897 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276594339}) 2024-09-16T03:18:04.897 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=359517919}) 2024-09-16T03:18:04.897 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.897 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276594339}) 2024-09-16T03:18:04.897 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.893+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3970124220}) 2024-09-16T03:18:04.898 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.898 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945246352}) 2024-09-16T03:18:04.898 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2284419410}) 2024-09-16T03:18:04.898 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.899 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945246352}) 2024-09-16T03:18:04.899 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2017529875}) 2024-09-16T03:18:04.900 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.900 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167342923}) 2024-09-16T03:18:04.900 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3243755864}) 2024-09-16T03:18:04.900 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.900 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167342923}) 2024-09-16T03:18:04.900 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2182655191}) 2024-09-16T03:18:04.901 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.901 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956170798}) 2024-09-16T03:18:04.901 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480429962}) 2024-09-16T03:18:04.901 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.902 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956170798}) 2024-09-16T03:18:04.902 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.897+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3452550545}) 2024-09-16T03:18:04.903 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.903 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610691815}) 2024-09-16T03:18:04.903 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1746024079}) 2024-09-16T03:18:04.903 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.903 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610691815}) 2024-09-16T03:18:04.904 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2741974440}) 2024-09-16T03:18:04.904 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.905 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=241267710}) 2024-09-16T03:18:04.905 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=726006247}) 2024-09-16T03:18:04.905 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.905 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=241267710}) 2024-09-16T03:18:04.905 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4193314329}) 2024-09-16T03:18:04.905 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.906 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410365218}) 2024-09-16T03:18:04.906 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3111092830}) 2024-09-16T03:18:04.906 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.906 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410365218}) 2024-09-16T03:18:04.906 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.901+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3835962410}) 2024-09-16T03:18:04.907 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.907 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156563884}) 2024-09-16T03:18:04.907 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3301025759}) 2024-09-16T03:18:04.907 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.907 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156563884}) 2024-09-16T03:18:04.907 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2731806401}) 2024-09-16T03:18:04.908 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.908 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447113001}) 2024-09-16T03:18:04.908 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=434703177}) 2024-09-16T03:18:04.908 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.909 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447113001}) 2024-09-16T03:18:04.909 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.905+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1557688792}) 2024-09-16T03:18:04.910 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.910 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2352776637}) 2024-09-16T03:18:04.910 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3552279847}) 2024-09-16T03:18:04.910 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.910 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2352776637}) 2024-09-16T03:18:04.910 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=232593961}) 2024-09-16T03:18:04.911 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.912 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847647999}) 2024-09-16T03:18:04.912 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2902452041}) 2024-09-16T03:18:04.912 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.912 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847647999}) 2024-09-16T03:18:04.912 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966628375}) 2024-09-16T03:18:04.913 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.913 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097310463}) 2024-09-16T03:18:04.913 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1442656148}) 2024-09-16T03:18:04.913 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.913 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097310463}) 2024-09-16T03:18:04.913 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.909+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2799683293}) 2024-09-16T03:18:04.914 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.914 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661498188}) 2024-09-16T03:18:04.915 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=345630352}) 2024-09-16T03:18:04.915 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.915 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661498188}) 2024-09-16T03:18:04.915 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3583257034}) 2024-09-16T03:18:04.916 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.916 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062455078}) 2024-09-16T03:18:04.916 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258823254}) 2024-09-16T03:18:04.917 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.917 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062455078}) 2024-09-16T03:18:04.917 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.913+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=15716305}) 2024-09-16T03:18:04.918 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.918 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390265679}) 2024-09-16T03:18:04.918 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=387952340}) 2024-09-16T03:18:04.918 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.918 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390265679}) 2024-09-16T03:18:04.918 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1990355462}) 2024-09-16T03:18:04.919 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.919 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478120483}) 2024-09-16T03:18:04.919 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2721373698}) 2024-09-16T03:18:04.919 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.919 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478120483}) 2024-09-16T03:18:04.919 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922669128}) 2024-09-16T03:18:04.921 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.921 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528121371}) 2024-09-16T03:18:04.921 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967239512}) 2024-09-16T03:18:04.921 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.921 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528121371}) 2024-09-16T03:18:04.921 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.917+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=394887312}) 2024-09-16T03:18:04.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130859554}) 2024-09-16T03:18:04.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=676942854}) 2024-09-16T03:18:04.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130859554}) 2024-09-16T03:18:04.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1478509129}) 2024-09-16T03:18:04.924 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.924 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665149869}) 2024-09-16T03:18:04.924 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4065078010}) 2024-09-16T03:18:04.924 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.924 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665149869}) 2024-09-16T03:18:04.924 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1671705435}) 2024-09-16T03:18:04.926 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.926 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=370963367}) 2024-09-16T03:18:04.926 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509960346}) 2024-09-16T03:18:04.926 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.921+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.926 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=370963367}) 2024-09-16T03:18:04.926 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1919081261}) 2024-09-16T03:18:04.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432780364}) 2024-09-16T03:18:04.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775396080}) 2024-09-16T03:18:04.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432780364}) 2024-09-16T03:18:04.928 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=233476528}) 2024-09-16T03:18:04.929 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.929 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915214710}) 2024-09-16T03:18:04.929 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3978725898}) 2024-09-16T03:18:04.929 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.929 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915214710}) 2024-09-16T03:18:04.930 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.925+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159027106}) 2024-09-16T03:18:04.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4} crc {logm=123162981,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1945130037,mon_config_key=1379474760}) 2024-09-16T03:18:04.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5} crc {logm=3995070082,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=2919432186,mon_config_key=2841741214}) 2024-09-16T03:18:04.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4} crc {logm=123162981,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1945130037,mon_config_key=1379474760}) 2024-09-16T03:18:04.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4} crc {logm=4137279305,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1945130037,mon_config_key=3384073484}) 2024-09-16T03:18:04.933 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.933 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=39} crc {mon_config_key=3499091205,monmap=127340147,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1094689084,osd_snap=288887718,osdmap=3493789414}) 2024-09-16T03:18:04.933 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=40} crc {mon_config_key=3160574477,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1094689084,osd_snap=2375936713,osdmap=26398193}) 2024-09-16T03:18:04.933 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.933 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=39} crc {mon_config_key=3499091205,monmap=127340147,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1094689084,osd_snap=288887718,osdmap=3493789414}) 2024-09-16T03:18:04.933 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.929+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=40} crc {mon_config_key=3499091205,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1094689084,osd_snap=1477566394,osdmap=1581239333}) 2024-09-16T03:18:04.934 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.934 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1340560189}) 2024-09-16T03:18:04.934 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1511523709}) 2024-09-16T03:18:04.935 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.935 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1340560189}) 2024-09-16T03:18:04.935 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4098693816}) 2024-09-16T03:18:04.936 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.936 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2365247866}) 2024-09-16T03:18:04.936 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=247232477}) 2024-09-16T03:18:04.937 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.937 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2365247866}) 2024-09-16T03:18:04.937 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2199370936}) 2024-09-16T03:18:04.938 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.938 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2065970727}) 2024-09-16T03:18:04.938 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1668429762}) 2024-09-16T03:18:04.938 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.938 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2065970727}) 2024-09-16T03:18:04.938 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.933+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3910423092}) 2024-09-16T03:18:04.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3373026176}) 2024-09-16T03:18:04.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3730531767}) 2024-09-16T03:18:04.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.939 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3373026176}) 2024-09-16T03:18:04.940 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2150630323}) 2024-09-16T03:18:04.940 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1142300319}) 2024-09-16T03:18:04.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=156607195}) 2024-09-16T03:18:04.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1142300319}) 2024-09-16T03:18:04.941 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.937+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3588793737}) 2024-09-16T03:18:04.942 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.941+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.942 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.941+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2419513529}) 2024-09-16T03:18:04.942 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.941+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=230690574}) 2024-09-16T03:18:04.943 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.941+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.943 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.941+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2419513529}) 2024-09-16T03:18:04.943 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.941+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1864383613}) 2024-09-16T03:18:04.947 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.945+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:18:04.947 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.945+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=2} crc {osdmap=3726878606}) 2024-09-16T03:18:04.947 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:18:04.945+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=1} crc {osdmap=627063106}) 2024-09-16T03:18:05.868 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:18:05.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient: get_monmap_and_config 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient: monmap: 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:06.002477+0000 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:06.002477+0000 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.001 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:06.001 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:06.001 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 1 Processor -- start 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 1 -- start start 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient: init 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient(hunting): picked mon.noname-c con 0x7f6680135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient(hunting): picked mon.noname-b con 0x7f668009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient(hunting): picked mon.noname-a con 0x7f668013c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f668009f7e0 con 0x7f668013c010 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f668009b160 con 0x7f6680135950 2024-09-16T03:18:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f668009b440 con 0x7f668009fc40 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:05.997+0000 7f668755b640 10 monclient(hunting): authenticate will time out at 1776.651955s 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6684af6640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f668009fc40 0x7f668013a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39100/0 (socket says 172.21.15.45:39100) 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6684af6640 1 -- 172.21.15.45:0/2229077158 learned_addr learned my addr 172.21.15.45:0/2229077158 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744142976 0 0) 0x7f668009f7e0 con 0x7f668013c010 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient(hunting): my global_id is 134355 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f666c003530 con 0x7f668013c010 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2683271458 0 0) 0x7f668009b440 con 0x7f668009fc40 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient(hunting): my global_id is 144251 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f668009f7e0 con 0x7f668009fc40 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2433848239 0 0) 0x7f666c003530 con 0x7f668013c010 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 >> v1:172.21.15.45:6790/0 conn(0x7f668009fc40 legacy=0x7f668013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 >> v1:172.21.15.46:6789/0 conn(0x7f6680135950 legacy=0x7f6680135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 monclient: found mon.noname-a 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6680136440 con 0x7f668013c010 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6678001a20 con 0x7f668013c010 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 5 monclient: authenticate success, global_id 134355 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: dump: 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.007 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6678001c40 con 0x7f668013c010 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 1 -- 172.21.15.45:0/2229077158 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6678002090 con 0x7f668013c010 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 4 set_mon_vals no callback set 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6677fff640 10 monclient: dump: 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient: shutdown 2024-09-16T03:18:06.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 -- 172.21.15.45:0/2229077158 >> v1:172.21.15.45:6789/0 conn(0x7f668013c010 legacy=0x7f668013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 -- 172.21.15.45:0/2229077158 shutdown_connections 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 -- 172.21.15.45:0/2229077158 >> 172.21.15.45:0/2229077158 conn(0x7f66800a2bd0 msgr2=0x7f66800a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 -- 172.21.15.45:0/2229077158 shutdown_connections 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 -- 172.21.15.45:0/2229077158 wait complete. 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient: monmap: 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:06.010 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 librados: starting msgr at 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 librados: starting objecter 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 Processor -- start 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 -- start start 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 librados: setting wanted keys 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 librados: calling monclient init 2024-09-16T03:18:06.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient: init 2024-09-16T03:18:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:18:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient(hunting): picked mon.noname-a con 0x7f668009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient(hunting): picked mon.noname-b con 0x7f6680135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient(hunting): picked mon.noname-c con 0x7f668013c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66800716c0 con 0x7f668009fc40 2024-09-16T03:18:06.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66800719a0 con 0x7f6680135950 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f668006f1d0 con 0x7f668013c010 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f668755b640 10 monclient(hunting): authenticate will time out at 1776.654958s 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6685af8640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f668013c010 0x7f66801d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39104/0 (socket says 172.21.15.45:39104) 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f66852f7640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f668009fc40 0x7f668006e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44240/0 (socket says 172.21.15.45:44240) 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6685af8640 1 -- 172.21.15.45:0/976505656 learned_addr learned my addr 172.21.15.45:0/976505656 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 client.?.objecter ms_handle_connect 0x7f668013c010 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 client.?.objecter ms_handle_connect 0x7f668009fc40 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2426507777 0 0) 0x7f668006f1d0 con 0x7f668013c010 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): my global_id is 144254 2024-09-16T03:18:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6664004ef0 con 0x7f668013c010 2024-09-16T03:18:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4124967081 0 0) 0x7f66800716c0 con 0x7f668009fc40 2024-09-16T03:18:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): my global_id is 134358 2024-09-16T03:18:06.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f668006f1d0 con 0x7f668009fc40 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 client.?.objecter ms_handle_connect 0x7f6680135950 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2612199703 0 0) 0x7f6664004ef0 con 0x7f668013c010 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66800716c0 con 0x7f668013c010 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6678001f90 con 0x7f668013c010 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): dump: 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.018 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.019 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2902567170 0 0) 0x7f668006f1d0 con 0x7f668009fc40 2024-09-16T03:18:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6664004ef0 con 0x7f668009fc40 2024-09-16T03:18:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 174261931 0 0) 0x7f66800719a0 con 0x7f6680135950 2024-09-16T03:18:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): my global_id is 154126 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f668006f1d0 con 0x7f6680135950 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f66700018c0 con 0x7f668009fc40 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient(hunting): dump: 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.020 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4124840246 0 0) 0x7f66800716c0 con 0x7f668013c010 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 >> v1:172.21.15.46:6789/0 conn(0x7f6680135950 legacy=0x7f668006eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 >> v1:172.21.15.45:6789/0 conn(0x7f668009fc40 legacy=0x7f668006e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 monclient: found mon.c 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.001+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668013b6b0 con 0x7f668013c010 2024-09-16T03:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 5 monclient: authenticate success, global_id 144254 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 10 monclient: _renew_subs 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f668006f640 con 0x7f668013c010 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 10 client.144254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 10 monclient: _renew_subs 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6680071100 con 0x7f668013c010 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 1 librados: init done 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6678001540 con 0x7f668013c010 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6678003f10 con 0x7f668013c010 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 10 monclient: dump: 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6676ffd640 4 set_mon_vals no callback set 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6660005090 con 0x7f668013c010 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f66780040b0 con 0x7f668013c010 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(537..537 src has 99..537) ==== 5488+0+0 (unknown 650750919 0 0) 0x7f667804bac0 con 0x7f668013c010 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 10 client.144254.objecter ms_dispatch 0x7f6680000ba0 osd_map(537..537 src has 99..537) 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 3 client.144254.objecter handle_osd_map got epochs [537,537] > 0 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 3 client.144254.objecter handle_osd_map decoding full epoch 537 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 20 client.144254.objecter dump_active .. 0 homeless 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f66852f7640 10 monclient: get_auth_request con 0x7f666403e290 auth_method 0 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.005+0000 7f6675ffb640 10 client.144254.objecter ms_handle_connect 0x7f666403e290 2024-09-16T03:18:06.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.009+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f6678016810 con 0x7f668013c010 2024-09-16T03:18:06.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.009+0000 7f6675ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.009+0000 7f6675ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.257+0000 7f668755b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:18:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.257+0000 7f668755b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:18:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.257+0000 7f668755b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.257+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6660005380 con 0x7f668013c010 2024-09-16T03:18:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.257+0000 7f6675ffb640 1 -- 172.21.15.45:0/976505656 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2593527600) 0x7f66780491a0 con 0x7f668013c010 2024-09-16T03:18:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.257+0000 7f6675ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:18:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.257+0000 7f6675ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:06.261 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:18:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 10 librados: watch_flush enter 2024-09-16T03:18:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 10 librados: watch_flush exit 2024-09-16T03:18:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 20 client.144254.objecter shutdown clearing up homeless session... 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 10 client.144254.objecter successfully canceled tick 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 >> v1:172.21.15.46:6812/149360280 conn(0x7f666403e290 legacy=0x7f6664040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 10 monclient: shutdown 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 >> v1:172.21.15.45:6790/0 conn(0x7f668013c010 legacy=0x7f66801d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 shutdown_connections 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 >> 172.21.15.45:0/976505656 conn(0x7f66800a2bd0 msgr2=0x7f668013eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 shutdown_connections 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 1 -- 172.21.15.45:0/976505656 wait complete. 2024-09-16T03:18:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.261+0000 7f668755b640 1 librados: shutdown 2024-09-16T03:18:06.279 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:18:06.279 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:18:06.279 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:18:06.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:18:06.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient: get_monmap_and_config 2024-09-16T03:18:06.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient: monmap: 2024-09-16T03:18:06.407 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.407 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.407 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:06.408839+0000 2024-09-16T03:18:06.407 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:06.408839+0000 2024-09-16T03:18:06.407 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.407 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.408 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.408 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:06.408 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:06.408 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 1 Processor -- start 2024-09-16T03:18:06.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 1 -- start start 2024-09-16T03:18:06.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient: init 2024-09-16T03:18:06.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient(hunting): picked mon.noname-b con 0x7fb13c135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient(hunting): picked mon.noname-a con 0x7fb13c09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient(hunting): picked mon.noname-c con 0x7fb13c13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb13c09f7e0 con 0x7fb13c09fc40 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb13c09b160 con 0x7fb13c13c010 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb13c09b440 con 0x7fb13c135950 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient(hunting): authenticate will time out at 1777.058679s 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb13c135950 0x7fb13c135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39110/0 (socket says 172.21.15.45:39110) 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13bfff640 1 -- 172.21.15.45:0/1272122895 learned_addr learned my addr 172.21.15.45:0/1272122895 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13b7fe640 1 --1- 172.21.15.45:0/1272122895 >> v1:172.21.15.45:6789/0 conn(0x7fb13c09fc40 0x7fb13c13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44252/0 (socket says 172.21.15.45:44252) 2024-09-16T03:18:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867872097 0 0) 0x7fb13c09b440 con 0x7fb13c135950 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): my global_id is 144257 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb124003530 con 0x7fb13c135950 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 722695335 0 0) 0x7fb13c09f7e0 con 0x7fb13c09fc40 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): my global_id is 134361 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb13c09b440 con 0x7fb13c09fc40 2024-09-16T03:18:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591586372 0 0) 0x7fb13c09b160 con 0x7fb13c13c010 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient(hunting): my global_id is 154129 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb13c09f7e0 con 0x7fb13c13c010 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2782099374 0 0) 0x7fb124003530 con 0x7fb13c135950 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 >> v1:172.21.15.46:6789/0 conn(0x7fb13c13c010 legacy=0x7fb13c13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 >> v1:172.21.15.45:6789/0 conn(0x7fb13c09fc40 legacy=0x7fb13c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 monclient: found mon.noname-b 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c136440 con 0x7fb13c135950 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb12c001620 con 0x7fb13c135950 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 5 monclient: authenticate success, global_id 144257 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: dump: 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb12c0017c0 con 0x7fb13c135950 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 1 -- 172.21.15.45:0/1272122895 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb12c001c50 con 0x7fb13c135950 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb1424cd640 4 set_mon_vals no callback set 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.405+0000 7fb13affd640 10 monclient: dump: 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: shutdown 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- 172.21.15.45:0/1272122895 >> v1:172.21.15.45:6790/0 conn(0x7fb13c135950 legacy=0x7fb13c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- 172.21.15.45:0/1272122895 shutdown_connections 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- 172.21.15.45:0/1272122895 >> 172.21.15.45:0/1272122895 conn(0x7fb13c0a2bd0 msgr2=0x7fb13c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- 172.21.15.45:0/1272122895 shutdown_connections 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- 172.21.15.45:0/1272122895 wait complete. 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: monmap: 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 librados: starting msgr at 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 librados: starting objecter 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 Processor -- start 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- start start 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 librados: setting wanted keys 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 librados: calling monclient init 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: init 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient(hunting): picked mon.noname-c con 0x7fb13c09fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient(hunting): picked mon.noname-b con 0x7fb13c135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient(hunting): picked mon.noname-a con 0x7fb13c13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb13c071280 con 0x7fb13c13c010 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb13c071560 con 0x7fb13c135950 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb13c071840 con 0x7fb13c09fc40 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient(hunting): authenticate will time out at 1777.062001s 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb13bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb13c09fc40 0x7fb13c06e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39118/0 (socket says 172.21.15.45:39118) 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb13bfff640 1 -- 172.21.15.45:0/3514346000 learned_addr learned my addr 172.21.15.45:0/3514346000 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 client.?.objecter ms_handle_connect 0x7fb13c09fc40 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 client.?.objecter ms_handle_connect 0x7fb13c13c010 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4047499179 0 0) 0x7fb13c071840 con 0x7fb13c09fc40 2024-09-16T03:18:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): my global_id is 144260 2024-09-16T03:18:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb128004ec0 con 0x7fb13c09fc40 2024-09-16T03:18:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 963928749 0 0) 0x7fb13c071280 con 0x7fb13c13c010 2024-09-16T03:18:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): my global_id is 134364 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb13c071840 con 0x7fb13c13c010 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 client.?.objecter ms_handle_connect 0x7fb13c135950 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 671034425 0 0) 0x7fb128004ec0 con 0x7fb13c09fc40 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb13c071280 con 0x7fb13c09fc40 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb12c001b10 con 0x7fb13c09fc40 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): dump: 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.426 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1981808552 0 0) 0x7fb13c071840 con 0x7fb13c13c010 2024-09-16T03:18:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb128004ec0 con 0x7fb13c13c010 2024-09-16T03:18:06.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 918149495 0 0) 0x7fb13c071560 con 0x7fb13c135950 2024-09-16T03:18:06.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): my global_id is 154132 2024-09-16T03:18:06.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb13c071840 con 0x7fb13c135950 2024-09-16T03:18:06.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb134001c00 con 0x7fb13c13c010 2024-09-16T03:18:06.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient(hunting): dump: 2024-09-16T03:18:06.429 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.429 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.429 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 582354789 0 0) 0x7fb13c071280 con 0x7fb13c09fc40 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 >> v1:172.21.15.46:6789/0 conn(0x7fb13c135950 legacy=0x7fb13c1ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 >> v1:172.21.15.45:6789/0 conn(0x7fb13c13c010 legacy=0x7fb13c1d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 monclient: found mon.c 2024-09-16T03:18:06.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c13b590 con 0x7fb13c09fc40 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 5 monclient: authenticate success, global_id 144260 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: _renew_subs 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c074100 con 0x7fb13c09fc40 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 client.144260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: _renew_subs 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb13c1d3b70 con 0x7fb13c09fc40 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 librados: init done 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb12c001720 con 0x7fb13c09fc40 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb12c004c00 con 0x7fb13c09fc40 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 10 monclient: dump: 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb139ffb640 4 set_mon_vals no callback set 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb11c0065d0 con 0x7fb11c002870 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.409+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fb12c010c70 con 0x7fb13c09fc40 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 client.144260.objecter ms_handle_connect 0x7fb11c002870 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 client.144260.objecter resend_mon_ops 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(538..538 src has 99..538) ==== 5861+0+0 (unknown 3834036138 0 0) 0x7fb12c04b9a0 con 0x7fb13c09fc40 2024-09-16T03:18:06.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 client.144260.objecter ms_dispatch 0x7fb13c000ba0 osd_map(538..538 src has 99..538) 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 3 client.144260.objecter handle_osd_map got epochs [538,538] > 0 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 3 client.144260.objecter handle_osd_map decoding full epoch 538 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 20 client.144260.objecter dump_active .. 0 homeless 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2239024877 0 0) 0x7fb11c0065d0 con 0x7fb11c002870 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 monclient(hunting): my global_id is 134367 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb13c071280 con 0x7fb11c002870 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb140a6a640 10 monclient: get_auth_request con 0x7fb12803e2b0 auth_method 0 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1203613647 0 0) 0x7fb13c071280 con 0x7fb11c002870 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb11c0065d0 con 0x7fb11c002870 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb1300017d0 con 0x7fb11c002870 2024-09-16T03:18:06.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 monclient: dump: 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 client.144260.objecter ms_handle_connect 0x7fb12803e2b0 2024-09-16T03:18:06.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2948765542 0 0) 0x7fb11c0065d0 con 0x7fb11c002870 2024-09-16T03:18:06.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb11c00a170 con 0x7fb11c002870 2024-09-16T03:18:06.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb130001a10 con 0x7fb11c002870 2024-09-16T03:18:06.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:06.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 >> v1:172.21.15.45:6789/0 conn(0x7fb11c002870 legacy=0x7fb11c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.413+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 >> v1:172.21.15.45:6789/0 conn(0x7fb11c002870 legacy=0x7fb11c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb1424cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb11c009e10 con 0x7fb11c0090f0 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb138ff9640 10 client.144260.objecter ms_handle_connect 0x7fb11c0090f0 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb138ff9640 10 client.144260.objecter resend_mon_ops 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423016956 0 0) 0x7fb11c009e10 con 0x7fb11c0090f0 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb138ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb138ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb138ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb138ff9640 10 monclient(hunting): my global_id is 134370 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.417+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb11c0065d0 con 0x7fb11c0090f0 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1874720107 0 0) 0x7fb11c0065d0 con 0x7fb11c0090f0 2024-09-16T03:18:06.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb11c009e10 con 0x7fb11c0090f0 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb130004990 con 0x7fb11c0090f0 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 10 monclient: dump: 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.439 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3241706865 0 0) 0x7fb11c009e10 con 0x7fb11c0090f0 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb11c00c3b0 con 0x7fb11c0090f0 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4304 (unknown 0 0 2250738922) 0x7fb130004920 con 0x7fb11c0090f0 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 >> v1:172.21.15.45:6789/0 conn(0x7fb11c0090f0 legacy=0x7fb11c009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb138ff9640 1 -- 172.21.15.45:0/3514346000 >> v1:172.21.15.45:6789/0 conn(0x7fb11c0090f0 legacy=0x7fb11c009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:18:06.440 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 56, 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 219649, 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:06.441 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:06.442 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:06.443 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:06.444 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:06.445 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:06.446 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:06.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.448 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:18:06.450 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.451 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.453 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:18:06.453 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:18:06.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 10 librados: watch_flush enter 2024-09-16T03:18:06.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 10 librados: watch_flush exit 2024-09-16T03:18:06.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 20 client.144260.objecter shutdown clearing up homeless session... 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 10 client.144260.objecter successfully canceled tick 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 >> v1:172.21.15.46:6812/149360280 conn(0x7fb12803e2b0 legacy=0x7fb128040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 10 monclient: shutdown 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 >> v1:172.21.15.45:6790/0 conn(0x7fb13c09fc40 legacy=0x7fb13c06e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb140a6a640 1 -- 172.21.15.45:0/3514346000 reap_dead start 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb140a6a640 1 -- 172.21.15.45:0/3514346000 reap_dead start 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 shutdown_connections 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 >> 172.21.15.45:0/3514346000 conn(0x7fb13c0a2bd0 msgr2=0x7fb13c0a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 shutdown_connections 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 1 -- 172.21.15.45:0/3514346000 wait complete. 2024-09-16T03:18:06.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.421+0000 7fb1424cd640 1 librados: shutdown 2024-09-16T03:18:06.456 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:18:06.456 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:18:06.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:18:06.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient: get_monmap_and_config 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient: monmap: 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:06.573233+0000 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:06.573233+0000 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:06.571 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 1 Processor -- start 2024-09-16T03:18:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 1 -- start start 2024-09-16T03:18:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient: init 2024-09-16T03:18:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:18:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5c137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5c138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5c13e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5c0a4f90 con 0x7f3c5c137d20 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5c0a4cb0 con 0x7f3c5c13e2a0 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5c09dc70 con 0x7f3c5c138100 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c6171c640 10 monclient(hunting): authenticate will time out at 1777.223286s 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3c5c137d20 0x7f3c5c0a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44284/0 (socket says 172.21.15.45:44284) 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c53fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3c5c138100 0x7f3c5c13cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39128/0 (socket says 172.21.15.45:39128) 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5affd640 1 -- 172.21.15.45:0/2268382891 learned_addr learned my addr 172.21.15.45:0/2268382891 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2460388449 0 0) 0x7f3c5c09dc70 con 0x7f3c5c138100 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 10 monclient(hunting): my global_id is 144263 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c48003530 con 0x7f3c5c138100 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056786107 0 0) 0x7f3c5c0a4f90 con 0x7f3c5c137d20 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 10 monclient(hunting): my global_id is 134373 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.569+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c5c09dc70 con 0x7f3c5c137d20 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525613812 0 0) 0x7f3c5c0a4cb0 con 0x7f3c5c13e2a0 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient(hunting): my global_id is 154135 2024-09-16T03:18:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c5c0a4f90 con 0x7f3c5c13e2a0 2024-09-16T03:18:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 646611221 0 0) 0x7f3c5c09dc70 con 0x7f3c5c137d20 2024-09-16T03:18:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 >> v1:172.21.15.45:6790/0 conn(0x7f3c5c138100 legacy=0x7f3c5c13cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 >> v1:172.21.15.46:6789/0 conn(0x7f3c5c13e2a0 legacy=0x7f3c5c140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 monclient: found mon.noname-a 2024-09-16T03:18:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:18:06.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c09cc50 con 0x7f3c5c137d20 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3c44001680 con 0x7f3c5c137d20 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: dump: 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.584 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 5 monclient: authenticate success, global_id 134373 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c44001820 con 0x7f3c5c137d20 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 1 -- 172.21.15.45:0/2268382891 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3c44001cb0 con 0x7f3c5c137d20 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 4 set_mon_vals no callback set 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5a7fc640 10 monclient: dump: 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: shutdown 2024-09-16T03:18:06.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- 172.21.15.45:0/2268382891 >> v1:172.21.15.45:6789/0 conn(0x7f3c5c137d20 legacy=0x7f3c5c0a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- 172.21.15.45:0/2268382891 shutdown_connections 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- 172.21.15.45:0/2268382891 >> 172.21.15.45:0/2268382891 conn(0x7f3c5c0aa110 msgr2=0x7f3c5c0aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- 172.21.15.45:0/2268382891 shutdown_connections 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- 172.21.15.45:0/2268382891 wait complete. 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: monmap: 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.587 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 librados: starting msgr at 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 librados: starting objecter 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 Processor -- start 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- start start 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 librados: setting wanted keys 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 librados: calling monclient init 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: init 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5c137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5c138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5c13e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5c139740 con 0x7f3c5c13e2a0 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5c141b10 con 0x7f3c5c138100 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5c0a1300 con 0x7f3c5c137d20 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient(hunting): authenticate will time out at 1777.226669s 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5affd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3c5c137d20 0x7f3c5c138920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39130/0 (socket says 172.21.15.45:39130) 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c5affd640 1 -- 172.21.15.45:0/3660483150 learned_addr learned my addr 172.21.15.45:0/3660483150 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 client.?.objecter ms_handle_connect 0x7f3c5c137d20 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 client.?.objecter ms_handle_connect 0x7f3c5c13e2a0 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523421418 0 0) 0x7f3c5c139740 con 0x7f3c5c13e2a0 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): my global_id is 134376 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c3c004ef0 con 0x7f3c5c13e2a0 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3519680644 0 0) 0x7f3c5c0a1300 con 0x7f3c5c137d20 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): my global_id is 144266 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c5c139740 con 0x7f3c5c137d20 2024-09-16T03:18:06.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 client.?.objecter ms_handle_connect 0x7f3c5c138100 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 68648378 0 0) 0x7f3c5c139740 con 0x7f3c5c137d20 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c5c0a1300 con 0x7f3c5c137d20 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3c44001bb0 con 0x7f3c5c137d20 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): dump: 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3714098697 0 0) 0x7f3c3c004ef0 con 0x7f3c5c13e2a0 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c5c139740 con 0x7f3c5c13e2a0 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3c54001c00 con 0x7f3c5c13e2a0 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): dump: 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155916446 0 0) 0x7f3c5c141b10 con 0x7f3c5c138100 2024-09-16T03:18:06.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient(hunting): my global_id is 154138 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c3c004ef0 con 0x7f3c5c138100 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2724745257 0 0) 0x7f3c5c0a1300 con 0x7f3c5c137d20 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 >> v1:172.21.15.46:6789/0 conn(0x7f3c5c138100 legacy=0x7f3c5c139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 >> v1:172.21.15.45:6789/0 conn(0x7f3c5c13e2a0 legacy=0x7f3c5c1d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 monclient: found mon.c 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c0a4430 con 0x7f3c5c137d20 2024-09-16T03:18:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 5 monclient: authenticate success, global_id 144266 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: _renew_subs 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5c13bfb0 con 0x7f3c5c137d20 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 client.144266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: _renew_subs 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3c5c13ab50 con 0x7f3c5c137d20 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c6171c640 1 librados: init done 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c44001780 con 0x7f3c5c137d20 2024-09-16T03:18:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3c44004da0 con 0x7f3c5c137d20 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c59ffb640 4 set_mon_vals no callback set 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.573+0000 7f3c537fe640 10 monclient: dump: 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c6171c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3c400065d0 con 0x7f3c40002870 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 client.144266.objecter ms_handle_connect 0x7f3c40002870 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 client.144266.objecter resend_mon_ops 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f3c44010e10 con 0x7f3c5c137d20 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(538..538 src has 99..538) ==== 5861+0+0 (unknown 3834036138 0 0) 0x7f3c4404ba00 con 0x7f3c5c137d20 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 client.144266.objecter ms_dispatch 0x7f3c5c000ba0 osd_map(538..538 src has 99..538) 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 3 client.144266.objecter handle_osd_map got epochs [538,538] > 0 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 3 client.144266.objecter handle_osd_map decoding full epoch 538 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 20 client.144266.objecter dump_active .. 0 homeless 2024-09-16T03:18:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 719097822 0 0) 0x7f3c400065d0 con 0x7f3c40002870 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 monclient(hunting): my global_id is 144269 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c5c0a1300 con 0x7f3c40002870 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c5b7fe640 10 monclient: get_auth_request con 0x7f3c3c03e2c0 auth_method 0 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 537063013 0 0) 0x7f3c5c0a1300 con 0x7f3c40002870 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c400065d0 con 0x7f3c40002870 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3c4c0012b0 con 0x7f3c40002870 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 monclient: dump: 2024-09-16T03:18:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4113324379 0 0) 0x7f3c400065d0 con 0x7f3c40002870 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c4000a170 con 0x7f3c40002870 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 client.144266.objecter ms_handle_connect 0x7f3c3c03e2c0 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3c4c001330 con 0x7f3c40002870 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:06.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 >> v1:172.21.15.45:6790/0 conn(0x7f3c40002870 legacy=0x7f3c40004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.577+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 >> v1:172.21.15.45:6790/0 conn(0x7f3c40002870 legacy=0x7f3c40004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:06.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.581+0000 7f3c6171c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:18:06.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.581+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3c40009e80 con 0x7f3c40009140 2024-09-16T03:18:06.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 client.144266.objecter ms_handle_connect 0x7f3c40009140 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 client.144266.objecter resend_mon_ops 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 755779674 0 0) 0x7f3c40009e80 con 0x7f3c40009140 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 monclient(hunting): my global_id is 144272 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c400065d0 con 0x7f3c40009140 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 336607332 0 0) 0x7f3c400065d0 con 0x7f3c40009140 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c40009e80 con 0x7f3c40009140 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3c4c004990 con 0x7f3c40009140 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 monclient: dump: 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2924271892 0 0) 0x7f3c40009e80 con 0x7f3c40009140 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c4000c400 con 0x7f3c40009140 2024-09-16T03:18:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 1145799482) 0x7f3c4c004920 con 0x7f3c40009140 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 >> v1:172.21.15.45:6790/0 conn(0x7f3c40009140 legacy=0x7f3c400095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c537fe640 1 -- 172.21.15.45:0/3660483150 >> v1:172.21.15.45:6790/0 conn(0x7f3c40009140 legacy=0x7f3c400095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 56, 2024-09-16T03:18:06.604 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 111658, 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:18:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:18:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:18:06.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.610 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:06.611 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:06.612 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:06.613 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:18:06.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.616 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:18:06.616 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:18:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 10 librados: watch_flush enter 2024-09-16T03:18:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 10 librados: watch_flush exit 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 20 client.144266.objecter shutdown clearing up homeless session... 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 10 client.144266.objecter successfully canceled tick 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 >> v1:172.21.15.46:6812/149360280 conn(0x7f3c3c03e2c0 legacy=0x7f3c3c040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 10 monclient: shutdown 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 >> v1:172.21.15.45:6790/0 conn(0x7f3c5c137d20 legacy=0x7f3c5c138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c5b7fe640 1 -- 172.21.15.45:0/3660483150 reap_dead start 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c5b7fe640 1 -- 172.21.15.45:0/3660483150 reap_dead start 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 shutdown_connections 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 >> 172.21.15.45:0/3660483150 conn(0x7f3c5c0aa110 msgr2=0x7f3c5c0742f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 shutdown_connections 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 1 -- 172.21.15.45:0/3660483150 wait complete. 2024-09-16T03:18:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.585+0000 7f3c6171c640 1 librados: shutdown 2024-09-16T03:18:06.619 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:18:06.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:18:06.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:18:06.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient: get_monmap_and_config 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient: monmap: 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:06.737246+0000 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:06.737246+0000 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:06.735 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 1 Processor -- start 2024-09-16T03:18:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 1 -- start start 2024-09-16T03:18:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient: init 2024-09-16T03:18:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:18:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd4137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd4138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd40a4f90 con 0x7f6bd4137d20 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd40a4cb0 con 0x7f6bd413e2a0 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd409dc70 con 0x7f6bd4138100 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient(hunting): authenticate will time out at 1777.386591s 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd2ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6bd4137d20 0x7f6bd40a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44302/0 (socket says 172.21.15.45:44302) 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd27fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6bd4138100 0x7f6bd413cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39150/0 (socket says 172.21.15.45:39150) 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd2ffd640 1 -- 172.21.15.45:0/4017302515 learned_addr learned my addr 172.21.15.45:0/4017302515 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3764662643 0 0) 0x7f6bd40a4f90 con 0x7f6bd4137d20 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient(hunting): my global_id is 134379 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bbc003530 con 0x7f6bd4137d20 2024-09-16T03:18:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 54329135 0 0) 0x7f6bd409dc70 con 0x7f6bd4138100 2024-09-16T03:18:06.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient(hunting): my global_id is 144275 2024-09-16T03:18:06.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd40a4f90 con 0x7f6bd4138100 2024-09-16T03:18:06.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1954757828 0 0) 0x7f6bbc003530 con 0x7f6bd4137d20 2024-09-16T03:18:06.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 >> v1:172.21.15.45:6790/0 conn(0x7f6bd4138100 legacy=0x7f6bd413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 >> v1:172.21.15.46:6789/0 conn(0x7f6bd413e2a0 legacy=0x7f6bd4140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 monclient: found mon.noname-a 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd409cc50 con 0x7f6bd4137d20 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6bc80016a0 con 0x7f6bd4137d20 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 5 monclient: authenticate success, global_id 134379 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: dump: 2024-09-16T03:18:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bc8001840 con 0x7f6bd4137d20 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:06.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd9314640 4 set_mon_vals no callback set 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 1 -- 172.21.15.45:0/4017302515 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6bc8001cd0 con 0x7f6bd4137d20 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.733+0000 7f6bd1ffb640 10 monclient: dump: 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.749 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: shutdown 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- 172.21.15.45:0/4017302515 >> v1:172.21.15.45:6789/0 conn(0x7f6bd4137d20 legacy=0x7f6bd40a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- 172.21.15.45:0/4017302515 shutdown_connections 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- 172.21.15.45:0/4017302515 >> 172.21.15.45:0/4017302515 conn(0x7f6bd40aa110 msgr2=0x7f6bd40aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- 172.21.15.45:0/4017302515 shutdown_connections 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- 172.21.15.45:0/4017302515 wait complete. 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: monmap: 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 librados: starting msgr at 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 librados: starting objecter 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 Processor -- start 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- start start 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 librados: setting wanted keys 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 librados: calling monclient init 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: init 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd4137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd4138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd413b800 con 0x7f6bd4138100 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd4139740 con 0x7f6bd413e2a0 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd413a2e0 con 0x7f6bd4137d20 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient(hunting): authenticate will time out at 1777.389902s 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd2ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6bd4137d20 0x7f6bd4138920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39152/0 (socket says 172.21.15.45:39152) 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd2ffd640 1 -- 172.21.15.45:0/1202577002 learned_addr learned my addr 172.21.15.45:0/1202577002 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 client.?.objecter ms_handle_connect 0x7f6bd4138100 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 client.?.objecter ms_handle_connect 0x7f6bd4137d20 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2736314114 0 0) 0x7f6bd413a2e0 con 0x7f6bd4137d20 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): my global_id is 144278 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bc0004ec0 con 0x7f6bd4137d20 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3557099537 0 0) 0x7f6bd413b800 con 0x7f6bd4138100 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): my global_id is 134382 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd413a2e0 con 0x7f6bd4138100 2024-09-16T03:18:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 client.?.objecter ms_handle_connect 0x7f6bd413e2a0 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3451230519 0 0) 0x7f6bc0004ec0 con 0x7f6bd4137d20 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bd413b800 con 0x7f6bd4137d20 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6bc8001b90 con 0x7f6bd4137d20 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): dump: 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.754 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 301746472 0 0) 0x7f6bd413a2e0 con 0x7f6bd4138100 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bc0004ec0 con 0x7f6bd4138100 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6bc40012e0 con 0x7f6bd4138100 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): dump: 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2231306732 0 0) 0x7f6bd4139740 con 0x7f6bd413e2a0 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient(hunting): my global_id is 154144 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd413a2e0 con 0x7f6bd413e2a0 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2829043857 0 0) 0x7f6bd413b800 con 0x7f6bd4137d20 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 >> v1:172.21.15.46:6789/0 conn(0x7f6bd413e2a0 legacy=0x7f6bd41d4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 >> v1:172.21.15.45:6789/0 conn(0x7f6bd4138100 legacy=0x7f6bd4139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 monclient: found mon.c 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd40a1300 con 0x7f6bd4137d20 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 5 monclient: authenticate success, global_id 144278 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: _renew_subs 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd4141690 con 0x7f6bd4137d20 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 client.144278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:06.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: _renew_subs 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6bd4140d70 con 0x7f6bd4137d20 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 librados: init done 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bc8004c00 con 0x7f6bd4137d20 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd17fa640 4 set_mon_vals no callback set 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6bc8004db0 con 0x7f6bd4137d20 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bbb7fe640 10 monclient: dump: 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.758 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.737+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6bb40065d0 con 0x7f6bb4002870 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f6bc8010e20 con 0x7f6bd4137d20 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(538..538 src has 99..538) ==== 5861+0+0 (unknown 3834036138 0 0) 0x7f6bc804b9c0 con 0x7f6bd4137d20 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 client.144278.objecter ms_dispatch 0x7f6bd4000ba0 osd_map(538..538 src has 99..538) 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 3 client.144278.objecter handle_osd_map got epochs [538,538] > 0 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 3 client.144278.objecter handle_osd_map decoding full epoch 538 2024-09-16T03:18:06.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 20 client.144278.objecter dump_active .. 0 homeless 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 client.144278.objecter ms_handle_connect 0x7f6bb4002870 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 client.144278.objecter resend_mon_ops 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bd37fe640 10 monclient: get_auth_request con 0x7f6bc003e300 auth_method 0 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434432025 0 0) 0x7f6bb40065d0 con 0x7f6bb4002870 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 monclient(hunting): my global_id is 154147 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd413b800 con 0x7f6bb4002870 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 client.144278.objecter ms_handle_connect 0x7f6bc003e300 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2380931068 0 0) 0x7f6bd413b800 con 0x7f6bb4002870 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bb40065d0 con 0x7f6bb4002870 2024-09-16T03:18:06.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6bc4004990 con 0x7f6bb4002870 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 monclient: dump: 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3970760751 0 0) 0x7f6bb40065d0 con 0x7f6bb4002870 2024-09-16T03:18:06.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6bb400a170 con 0x7f6bb4002870 2024-09-16T03:18:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6bc4004920 con 0x7f6bb4002870 2024-09-16T03:18:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 >> v1:172.21.15.46:6789/0 conn(0x7f6bb4002870 legacy=0x7f6bb4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.741+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 >> v1:172.21.15.46:6789/0 conn(0x7f6bb4002870 legacy=0x7f6bb4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:18:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6bb4009e60 con 0x7f6bb4009120 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 client.144278.objecter ms_handle_connect 0x7f6bb4009120 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 client.144278.objecter resend_mon_ops 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 153099596 0 0) 0x7f6bb4009e60 con 0x7f6bb4009120 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 monclient(hunting): my global_id is 154150 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb40065d0 con 0x7f6bb4009120 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1561547747 0 0) 0x7f6bb40065d0 con 0x7f6bb4009120 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bb4009e60 con 0x7f6bb4009120 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6bc4004f30 con 0x7f6bb4009120 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 monclient: dump: 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1523563395 0 0) 0x7f6bb4009e60 con 0x7f6bb4009120 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6bb400c3b0 con 0x7f6bb4009120 2024-09-16T03:18:06.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3966 (unknown 0 0 1153846709) 0x7f6bc4006420 con 0x7f6bb4009120 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 >> v1:172.21.15.46:6789/0 conn(0x7f6bb4009120 legacy=0x7f6bb40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bbb7fe640 1 -- 172.21.15.45:0/1202577002 >> v1:172.21.15.46:6789/0 conn(0x7f6bb4009120 legacy=0x7f6bb40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 56, 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 3079, 2024-09-16T03:18:06.766 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:06.767 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:06.768 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:18:06.769 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:18:06.770 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:18:06.770 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:18:06.770 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:06.770 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:06.772 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:18:06.772 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:18:06.772 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:18:06.772 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:18:06.772 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:18:06.772 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:18:06.772 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:18:06.772 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:06.772 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:18:06.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.774 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:06.775 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:06.776 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:06.777 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:06.778 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:18:06.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:06.779 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:06.779 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:06.779 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:06.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:06.779 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:06.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:06.779 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:18:06.779 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:18:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 10 librados: watch_flush enter 2024-09-16T03:18:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 10 librados: watch_flush exit 2024-09-16T03:18:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 20 client.144278.objecter shutdown clearing up homeless session... 2024-09-16T03:18:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 10 client.144278.objecter successfully canceled tick 2024-09-16T03:18:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 >> v1:172.21.15.46:6812/149360280 conn(0x7f6bc003e300 legacy=0x7f6bc00407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 10 monclient: shutdown 2024-09-16T03:18:06.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 >> v1:172.21.15.45:6790/0 conn(0x7f6bd4137d20 legacy=0x7f6bd4138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd37fe640 1 -- 172.21.15.45:0/1202577002 reap_dead start 2024-09-16T03:18:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd37fe640 1 -- 172.21.15.45:0/1202577002 reap_dead start 2024-09-16T03:18:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 shutdown_connections 2024-09-16T03:18:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 >> 172.21.15.45:0/1202577002 conn(0x7f6bd40aa110 msgr2=0x7f6bd40a0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 shutdown_connections 2024-09-16T03:18:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 1 -- 172.21.15.45:0/1202577002 wait complete. 2024-09-16T03:18:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.749+0000 7f6bd9314640 1 librados: shutdown 2024-09-16T03:18:06.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:18:06.783 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:18:06.783 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T03:18:06.783 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:18:06.783 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-16T03:18:06.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-16T03:18:06.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.917+0000 7f34b4a44640 10 monclient: get_monmap_and_config 2024-09-16T03:18:06.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.917+0000 7f34b4a44640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.917+0000 7f34b4a44640 10 monclient: monmap: 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:06.923985+0000 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:06.923985+0000 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:06.922 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 Processor -- start 2024-09-16T03:18:06.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 -- start start 2024-09-16T03:18:06.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient: init 2024-09-16T03:18:06.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:18:06.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient(hunting): picked mon.noname-b con 0x7f34b0076580 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient(hunting): picked mon.noname-a con 0x7f34b009faf0 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient(hunting): picked mon.noname-c con 0x7f34b013afc0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b00a2fe0 con 0x7f34b009faf0 2024-09-16T03:18:06.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b00a0e80 con 0x7f34b013afc0 2024-09-16T03:18:06.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b009eda0 con 0x7f34b0076580 2024-09-16T03:18:06.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient(hunting): authenticate will time out at 1777.573327s 2024-09-16T03:18:06.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ae59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f34b0076580 0x7f34b00a21c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39156/0 (socket says 172.21.15.45:39156) 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ae59c640 1 -- 172.21.15.45:0/4249986724 learned_addr learned my addr 172.21.15.45:0/4249986724 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34add9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f34b009faf0 0x7f34b00a28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44310/0 (socket says 172.21.15.45:44310) 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 257470506 0 0) 0x7f34b009eda0 con 0x7f34b0076580 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): my global_id is 144281 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3498003530 con 0x7f34b0076580 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460912888 0 0) 0x7f34b00a2fe0 con 0x7f34b009faf0 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): my global_id is 134385 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34b009eda0 con 0x7f34b009faf0 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875417479 0 0) 0x7f34b00a0e80 con 0x7f34b013afc0 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient(hunting): my global_id is 154153 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34b00a2fe0 con 0x7f34b013afc0 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1507981067 0 0) 0x7f3498003530 con 0x7f34b0076580 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 >> v1:172.21.15.46:6789/0 conn(0x7f34b013afc0 legacy=0x7f34b013d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 >> v1:172.21.15.45:6789/0 conn(0x7f34b009faf0 legacy=0x7f34b00a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 monclient: found mon.noname-b 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b009f080 con 0x7f34b0076580 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 5 monclient: authenticate success, global_id 144281 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f34a00016a0 con 0x7f34b0076580 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: dump: 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.934 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f34a0001840 con 0x7f34b0076580 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 1 -- 172.21.15.45:0/4249986724 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f34a0001c50 con 0x7f34b0076580 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 4 set_mon_vals no callback set 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34ad59a640 10 monclient: dump: 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.936 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient: shutdown 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 -- 172.21.15.45:0/4249986724 >> v1:172.21.15.45:6790/0 conn(0x7f34b0076580 legacy=0x7f34b00a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 -- 172.21.15.45:0/4249986724 shutdown_connections 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 -- 172.21.15.45:0/4249986724 >> 172.21.15.45:0/4249986724 conn(0x7f34b00a6de0 msgr2=0x7f34b00a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 -- 172.21.15.45:0/4249986724 shutdown_connections 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 -- 172.21.15.45:0/4249986724 wait complete. 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient: build_initial_monmap 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 10 monclient: monmap: 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:06.937 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 librados: starting msgr at 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.921+0000 7f34b4a44640 1 librados: starting objecter 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 Processor -- start 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 -- start start 2024-09-16T03:18:06.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 librados: setting wanted keys 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 librados: calling monclient init 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient: init 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient(hunting): picked mon.noname-b con 0x7f34b0076580 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient(hunting): picked mon.noname-c con 0x7f34b009faf0 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient(hunting): picked mon.noname-a con 0x7f34b013afc0 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b01387e0 con 0x7f34b013afc0 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b01380e0 con 0x7f34b0076580 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b01383c0 con 0x7f34b009faf0 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient(hunting): authenticate will time out at 1777.576628s 2024-09-16T03:18:06.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34add9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f34b009faf0 0x7f34b0135bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39162/0 (socket says 172.21.15.45:39162) 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34add9b640 1 -- 172.21.15.45:0/3389918747 learned_addr learned my addr 172.21.15.45:0/3389918747 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34aed9d640 1 --1- 172.21.15.45:0/3389918747 >> v1:172.21.15.45:6789/0 conn(0x7f34b013afc0 0x7f34b01d2af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44314/0 (socket says 172.21.15.45:44314) 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.?.objecter ms_handle_connect 0x7f34b009faf0 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.?.objecter ms_handle_connect 0x7f34b013afc0 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378531102 0 0) 0x7f34b01383c0 con 0x7f34b009faf0 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): my global_id is 144284 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3490004ec0 con 0x7f34b009faf0 2024-09-16T03:18:06.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714401353 0 0) 0x7f34b00a2fe0 con 0x7f34b013afc0 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): my global_id is 134388 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34b01383c0 con 0x7f34b013afc0 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.?.objecter ms_handle_connect 0x7f34b0076580 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 647274274 0 0) 0x7f3490004ec0 con 0x7f34b009faf0 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34b00a2fe0 con 0x7f34b009faf0 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f34a40013a0 con 0x7f34b009faf0 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:06.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): dump: 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3316245045 0 0) 0x7f34b01383c0 con 0x7f34b013afc0 2024-09-16T03:18:06.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3490004ec0 con 0x7f34b013afc0 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f34a8001bc0 con 0x7f34b013afc0 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): dump: 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.943 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2676420639 0 0) 0x7f34b01380e0 con 0x7f34b0076580 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): my global_id is 154156 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34b01383c0 con 0x7f34b0076580 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3201685440 0 0) 0x7f34b00a2fe0 con 0x7f34b009faf0 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 >> v1:172.21.15.46:6789/0 conn(0x7f34b0076580 legacy=0x7f34b01354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 >> v1:172.21.15.45:6789/0 conn(0x7f34b013afc0 legacy=0x7f34b01d2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: _finish_hunting 0 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 monclient: found mon.c 2024-09-16T03:18:06.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b01362d0 con 0x7f34b009faf0 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: _finish_auth 0 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: _check_auth_tickets 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 5 monclient: authenticate success, global_id 144284 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient: _renew_subs 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f34b0074cb0 con 0x7f34b009faf0 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 client.144284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient: _renew_subs 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:06.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f34b0073840 con 0x7f34b009faf0 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 librados: init done 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f34a40047f0 con 0x7f34b009faf0 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f34a4004bc0 con 0x7f34b009faf0 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34acd99640 4 set_mon_vals no callback set 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: dump: 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f34940065d0 con 0x7f3494002870 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f34a4010cb0 con 0x7f34b009faf0 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.144284.objecter ms_handle_connect 0x7f3494002870 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.144284.objecter resend_mon_ops 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(539..539 src has 99..539) ==== 5874+0+0 (unknown 909202506 0 0) 0x7f34a404b9f0 con 0x7f34b009faf0 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.144284.objecter ms_dispatch 0x7f34b0000ba0 osd_map(539..539 src has 99..539) 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 3 client.144284.objecter handle_osd_map got epochs [539,539] > 0 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 3 client.144284.objecter handle_osd_map decoding full epoch 539 2024-09-16T03:18:06.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 20 client.144284.objecter dump_active .. 0 homeless 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1149654533 0 0) 0x7f34940065d0 con 0x7f3494002870 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient(hunting): my global_id is 154159 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34b00a2fe0 con 0x7f3494002870 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f34aed9d640 10 monclient: get_auth_request con 0x7f349003e2b0 auth_method 0 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2596499196 0 0) 0x7f34b00a2fe0 con 0x7f3494002870 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34940065d0 con 0x7f3494002870 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f34a0001760 con 0x7f3494002870 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 monclient: dump: 2024-09-16T03:18:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4151781475 0 0) 0x7f34940065d0 con 0x7f3494002870 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f349400a170 con 0x7f3494002870 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.925+0000 7f349effd640 10 client.144284.objecter ms_handle_connect 0x7f349003e2b0 2024-09-16T03:18:06.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.929+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f34a00019e0 con 0x7f3494002870 2024-09-16T03:18:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.929+0000 7f349effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.929+0000 7f349effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.929+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 >> v1:172.21.15.46:6789/0 conn(0x7f3494002870 legacy=0x7f3494004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.929+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 >> v1:172.21.15.46:6789/0 conn(0x7f3494002870 legacy=0x7f3494004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.933+0000 7f34b4a44640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:18:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.933+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3494009ea0 con 0x7f3494009160 2024-09-16T03:18:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.933+0000 7f349effd640 10 client.144284.objecter ms_handle_connect 0x7f3494009160 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.933+0000 7f349effd640 10 client.144284.objecter resend_mon_ops 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4123722694 0 0) 0x7f3494009ea0 con 0x7f3494009160 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 10 monclient(hunting): my global_id is 154162 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34940065d0 con 0x7f3494009160 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3852404671 0 0) 0x7f34940065d0 con 0x7f3494009160 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3494009ea0 con 0x7f3494009160 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f34a0004af0 con 0x7f3494009160 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:18:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 10 monclient: dump: 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2042112710 0 0) 0x7f3494009ea0 con 0x7f3494009160 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f349400c400 con 0x7f3494009160 2024-09-16T03:18:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (unknown 0 0 2946029456) 0x7f34a0004ae0 con 0x7f3494009160 2024-09-16T03:18:06.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:18:06.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:06.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 >> v1:172.21.15.46:6789/0 conn(0x7f3494009160 legacy=0x7f34940095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f349effd640 1 -- 172.21.15.45:0/3389918747 >> v1:172.21.15.46:6789/0 conn(0x7f3494009160 legacy=0x7f34940095f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:06.954 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:18:06.954 INFO:teuthology.orchestra.run.smithi045.stdout: "ret": 0, 2024-09-16T03:18:06.954 INFO:teuthology.orchestra.run.smithi045.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T03:18:06.954 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:18:06.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 10 librados: watch_flush enter 2024-09-16T03:18:06.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 10 librados: watch_flush exit 2024-09-16T03:18:06.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 20 client.144284.objecter shutdown clearing up homeless session... 2024-09-16T03:18:06.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 10 client.144284.objecter successfully canceled tick 2024-09-16T03:18:06.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 >> v1:172.21.15.46:6812/149360280 conn(0x7f349003e2b0 legacy=0x7f3490040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 10 monclient: shutdown 2024-09-16T03:18:06.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:06.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 >> v1:172.21.15.45:6790/0 conn(0x7f34b009faf0 legacy=0x7f34b0135bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:06.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34aed9d640 1 -- 172.21.15.45:0/3389918747 reap_dead start 2024-09-16T03:18:06.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34aed9d640 1 -- 172.21.15.45:0/3389918747 reap_dead start 2024-09-16T03:18:06.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 shutdown_connections 2024-09-16T03:18:06.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 >> 172.21.15.45:0/3389918747 conn(0x7f34b00a6de0 msgr2=0x7f34b0134170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:06.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 shutdown_connections 2024-09-16T03:18:06.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 1 -- 172.21.15.45:0/3389918747 wait complete. 2024-09-16T03:18:06.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:06.937+0000 7f34b4a44640 1 librados: shutdown 2024-09-16T03:18:06.957 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:18:06.958 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:18:06.958 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:18:06.979 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:18:06.979 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:18:06.979 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:18:06.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:18:21.980 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:18:21.981 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:18:21.981 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:18:21.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:18:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient: get_monmap_and_config 2024-09-16T03:18:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient: build_initial_monmap 2024-09-16T03:18:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient: monmap: 2024-09-16T03:18:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:22.117477+0000 2024-09-16T03:18:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:22.117477+0000 2024-09-16T03:18:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:22.116 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 1 Processor -- start 2024-09-16T03:18:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 1 -- start start 2024-09-16T03:18:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient: init 2024-09-16T03:18:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:18:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient(hunting): picked mon.noname-c con 0x7f3004135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient(hunting): picked mon.noname-a con 0x7f300409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient(hunting): picked mon.noname-b con 0x7f300413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300409f7e0 con 0x7f300409fc40 2024-09-16T03:18:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300409b160 con 0x7f3004135950 2024-09-16T03:18:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300409b440 con 0x7f300413c010 2024-09-16T03:18:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30093b3640 10 monclient(hunting): authenticate will time out at 1792.766941s 2024-09-16T03:18:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f3002ffd640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f3004135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30027fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f300409fc40 0x7f300413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37512/0 (socket says 172.21.15.45:37512) 2024-09-16T03:18:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.112+0000 7f30027fc640 1 -- 172.21.15.45:0/702928791 learned_addr learned my addr 172.21.15.45:0/702928791 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:22.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.316+0000 7f3002ffd640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f3004135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:22.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:22.720+0000 7f3002ffd640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f3004135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:23.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:23.524+0000 7f3002ffd640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f3004135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 10 monclient(hunting): tick 2024-09-16T03:18:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:18:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 1 monclient(hunting): continuing hunt 2024-09-16T03:18:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:18:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.45:6790/0 conn(0x7f300413c010 legacy=0x7f300413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f3004135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:18:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.45:6789/0 conn(0x7f300409fc40 legacy=0x7f300413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:18:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f2fe40036d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f3004135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f2fe400ab70 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 1 -- 172.21.15.45:0/702928791 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3004136440 con 0x7f2fe400ab70 2024-09-16T03:18:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 1 -- 172.21.15.45:0/702928791 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300409b160 con 0x7f3004135950 2024-09-16T03:18:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 1 -- 172.21.15.45:0/702928791 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fe4010900 con 0x7f2fe40036d0 2024-09-16T03:18:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f3000ff9640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.112+0000 7f30027fc640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f2fe4009400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:25.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.316+0000 7f30027fc640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f2fe4009400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:25.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:25.720+0000 7f30027fc640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f2fe4009400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.524+0000 7f30027fc640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f2fe4009400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 -- 172.21.15.45:0/702928791 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122059151 0 0) 0x7f2fe4010900 con 0x7f2fe40036d0 2024-09-16T03:18:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:18:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient(hunting): my global_id is 144287 2024-09-16T03:18:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 -- 172.21.15.45:0/702928791 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ff0003530 con 0x7f2fe40036d0 2024-09-16T03:18:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 -- 172.21.15.45:0/702928791 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2087106270 0 0) 0x7f2ff0003530 con 0x7f2fe40036d0 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f2fe4009400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.45:6789/0 conn(0x7f2fe400ab70 legacy=0x7f2fe400d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 monclient: found mon.noname-b 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 -- 172.21.15.45:0/702928791 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe4011ba0 con 0x7f2fe40036d0 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: _finish_auth 0 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 -- 172.21.15.45:0/702928791 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2fec004230 con 0x7f2fe40036d0 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: dump: 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f30093b3640 5 monclient: authenticate success, global_id 144287 2024-09-16T03:18:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f30093b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 -- 172.21.15.45:0/702928791 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2fec0043d0 con 0x7f2fe40036d0 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 1 -- 172.21.15.45:0/702928791 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2fec004820 con 0x7f2fe40036d0 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f30093b3640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f30093b3640 4 set_mon_vals no callback set 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:26.996+0000 7f3001ffb640 10 monclient: dump: 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.003 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.004 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.004 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.004 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient: shutdown 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 -- 172.21.15.45:0/702928791 >> v1:172.21.15.45:6790/0 conn(0x7f2fe40036d0 legacy=0x7f2fe4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 -- 172.21.15.45:0/702928791 shutdown_connections 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 -- 172.21.15.45:0/702928791 >> 172.21.15.45:0/702928791 conn(0x7f30040a2bd0 msgr2=0x7f30040a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 -- 172.21.15.45:0/702928791 shutdown_connections 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 -- 172.21.15.45:0/702928791 wait complete. 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient: build_initial_monmap 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient: monmap: 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:27.008 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 librados: starting msgr at 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 librados: starting objecter 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 Processor -- start 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 -- start start 2024-09-16T03:18:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 librados: setting wanted keys 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 librados: calling monclient init 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient: init 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient(hunting): picked mon.noname-c con 0x7f2fe40036d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient(hunting): picked mon.noname-a con 0x7f2fe400ab70 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient(hunting): picked mon.noname-b con 0x7f3004135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3004073340 con 0x7f2fe400ab70 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300412cec0 con 0x7f3004135950 2024-09-16T03:18:27.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300413af60 con 0x7f2fe40036d0 2024-09-16T03:18:27.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:27.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30093b3640 10 monclient(hunting): authenticate will time out at 1797.655119s 2024-09-16T03:18:27.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30037fe640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f30041d48b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:27.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30027fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2fe400ab70 0x7f3004072c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37520/0 (socket says 172.21.15.45:37520) 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f30027fc640 1 -- 172.21.15.45:0/2346146356 learned_addr learned my addr 172.21.15.45:0/2346146356 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 client.?.objecter ms_handle_connect 0x7f2fe40036d0 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 client.?.objecter ms_handle_connect 0x7f2fe400ab70 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2590685941 0 0) 0x7f300413af60 con 0x7f2fe40036d0 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 monclient(hunting): my global_id is 144290 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fe4006b90 con 0x7f2fe40036d0 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1836164094 0 0) 0x7f2fe4006b90 con 0x7f2fe40036d0 2024-09-16T03:18:27.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f300413af60 con 0x7f2fe40036d0 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2fec004580 con 0x7f2fe40036d0 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.000+0000 7f2ffb7fe640 10 monclient(hunting): dump: 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3089300277 0 0) 0x7f300413af60 con 0x7f2fe40036d0 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 >> v1:172.21.15.46:6789/0 conn(0x7f3004135950 legacy=0x7f30041d48b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 >> v1:172.21.15.45:6789/0 conn(0x7f2fe400ab70 legacy=0x7f3004072c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 1 monclient: found mon.c 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300409d2c0 con 0x7f2fe40036d0 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 5 monclient: authenticate success, global_id 144290 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 10 monclient: _renew_subs 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f300412cbe0 con 0x7f2fe40036d0 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 10 client.144290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 10 monclient: _renew_subs 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f30041d61a0 con 0x7f2fe40036d0 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 1 librados: init done 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2fec004330 con 0x7f2fe40036d0 2024-09-16T03:18:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2fec004b70 con 0x7f2fe40036d0 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30017fa640 4 set_mon_vals no callback set 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 monclient: dump: 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.016 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fe8005090 con 0x7f2fe40036d0 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f2fec004d10 con 0x7f2fe40036d0 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(539..539 src has 99..539) ==== 5874+0+0 (unknown 909202506 0 0) 0x7f2fec04b880 con 0x7f2fe40036d0 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 client.144290.objecter ms_dispatch 0x7f3004000ba0 osd_map(539..539 src has 99..539) 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 3 client.144290.objecter handle_osd_map got epochs [539,539] > 0 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 3 client.144290.objecter handle_osd_map decoding full epoch 539 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 20 client.144290.objecter dump_active .. 0 homeless 2024-09-16T03:18:27.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f30027fc640 10 monclient: get_auth_request con 0x7f2fe403ee00 auth_method 0 2024-09-16T03:18:27.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.004+0000 7f2ffb7fe640 10 client.144290.objecter ms_handle_connect 0x7f2fe403ee00 2024-09-16T03:18:27.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.008+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f2fec0169d0 con 0x7f2fe40036d0 2024-09-16T03:18:27.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.008+0000 7f2ffb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:27.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.008+0000 7f2ffb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.256+0000 7f30093b3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:18:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.256+0000 7f30093b3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:18:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.256+0000 7f30093b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.256+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2fe8005380 con 0x7f2fe40036d0 2024-09-16T03:18:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.256+0000 7f2ffb7fe640 1 -- 172.21.15.45:0/2346146356 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 953382557) 0x7f2fec01fc30 con 0x7f2fe40036d0 2024-09-16T03:18:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.256+0000 7f2ffb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:18:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.256+0000 7f2ffb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:27.261 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":58,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 10 librados: watch_flush enter 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 10 librados: watch_flush exit 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 20 client.144290.objecter shutdown clearing up homeless session... 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 10 client.144290.objecter successfully canceled tick 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 >> v1:172.21.15.46:6812/149360280 conn(0x7f2fe403ee00 legacy=0x7f2fe40412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 10 monclient: shutdown 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 >> v1:172.21.15.45:6790/0 conn(0x7f2fe40036d0 legacy=0x7f3004072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 shutdown_connections 2024-09-16T03:18:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 >> 172.21.15.45:0/2346146356 conn(0x7f30040a2bd0 msgr2=0x7f30040a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 shutdown_connections 2024-09-16T03:18:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 1 -- 172.21.15.45:0/2346146356 wait complete. 2024-09-16T03:18:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.260+0000 7f30093b3640 1 librados: shutdown 2024-09-16T03:18:27.279 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:18:27.279 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:18:27.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:18:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient: get_monmap_and_config 2024-09-16T03:18:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient: build_initial_monmap 2024-09-16T03:18:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient: monmap: 2024-09-16T03:18:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:27.400775+0000 2024-09-16T03:18:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:27.400775+0000 2024-09-16T03:18:27.399 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:27.399 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.399 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:27.399 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:27.399 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:27.399 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 1 Processor -- start 2024-09-16T03:18:27.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 1 -- start start 2024-09-16T03:18:27.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient: init 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient(hunting): picked mon.noname-c con 0x7f755809b8e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient(hunting): picked mon.noname-a con 0x7f755809b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient(hunting): picked mon.noname-b con 0x7f755813b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755809cc20 con 0x7f755809b250 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755809d820 con 0x7f755809b8e0 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755809b6a0 con 0x7f755813b000 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient(hunting): authenticate will time out at 1798.050277s 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755659c640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f755809b8e0 legacy=0x7f755809bcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f7555d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f755809b250 0x7f75581398e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37528/0 (socket says 172.21.15.45:37528) 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f7555d9b640 1 -- 172.21.15.45:0/2278676719 learned_addr learned my addr 172.21.15.45:0/2278676719 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109418517 0 0) 0x7f755809cc20 con 0x7f755809b250 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient(hunting): my global_id is 134391 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7544003530 con 0x7f755809b250 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 759571030 0 0) 0x7f755809b6a0 con 0x7f755813b000 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient(hunting): my global_id is 144293 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f755809cc20 con 0x7f755813b000 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 344612673 0 0) 0x7f755809cc20 con 0x7f755813b000 2024-09-16T03:18:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 >> v1:172.21.15.46:6789/0 conn(0x7f755809b8e0 legacy=0x7f755809bcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 >> v1:172.21.15.45:6789/0 conn(0x7f755809b250 legacy=0x7f75581398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: _finish_hunting 0 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 monclient: found mon.noname-b 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75580a21f0 con 0x7f755813b000 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: _finish_auth 0 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: _check_auth_tickets 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7550001600 con 0x7f755813b000 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:18:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: dump: 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 5 monclient: authenticate success, global_id 144293 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f75500017a0 con 0x7f755813b000 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 4 set_mon_vals no callback set 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 1 -- 172.21.15.45:0/2278676719 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7550001bb0 con 0x7f755813b000 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755559a640 10 monclient: dump: 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 10 monclient: shutdown 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 1 -- 172.21.15.45:0/2278676719 >> v1:172.21.15.45:6790/0 conn(0x7f755813b000 legacy=0x7f755813d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 1 -- 172.21.15.45:0/2278676719 shutdown_connections 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.396+0000 7f755cb94640 1 -- 172.21.15.45:0/2278676719 >> 172.21.15.45:0/2278676719 conn(0x7f75580a2bd0 msgr2=0x7f75580a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 -- 172.21.15.45:0/2278676719 shutdown_connections 2024-09-16T03:18:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 -- 172.21.15.45:0/2278676719 wait complete. 2024-09-16T03:18:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient: build_initial_monmap 2024-09-16T03:18:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient: monmap: 2024-09-16T03:18:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 librados: starting msgr at 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 librados: starting objecter 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 Processor -- start 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 -- start start 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 librados: setting wanted keys 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 librados: calling monclient init 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient: init 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient(hunting): picked mon.noname-b con 0x7f755809b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient(hunting): picked mon.noname-c con 0x7f755809b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient(hunting): picked mon.noname-a con 0x7f755813b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755813a4a0 con 0x7f755813b000 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7558136d50 con 0x7f755809b250 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7558137030 con 0x7f755809b8e0 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient(hunting): authenticate will time out at 1798.053805s 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f7555d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f755809b8e0 0x7f755813cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:54648/0 (socket says 172.21.15.45:54648) 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755659c640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f755809b250 legacy=0x7f755813c830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f7555d9b640 1 -- 172.21.15.45:0/2371329530 learned_addr learned my addr 172.21.15.45:0/2371329530 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 client.?.objecter ms_handle_connect 0x7f755809b8e0 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 client.?.objecter ms_handle_connect 0x7f755813b000 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2312019577 0 0) 0x7f7558137030 con 0x7f755809b8e0 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): my global_id is 144296 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7538004ec0 con 0x7f755809b8e0 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211226862 0 0) 0x7f755813a4a0 con 0x7f755813b000 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): my global_id is 134394 2024-09-16T03:18:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7558137030 con 0x7f755813b000 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3332773407 0 0) 0x7f7538004ec0 con 0x7f755809b8e0 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f755813a4a0 con 0x7f755809b8e0 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7540001400 con 0x7f755809b8e0 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): dump: 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3987663941 0 0) 0x7f7558137030 con 0x7f755813b000 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7538004ec0 con 0x7f755813b000 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7550001af0 con 0x7f755813b000 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): dump: 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 986735183 0 0) 0x7f755813a4a0 con 0x7f755809b8e0 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 >> v1:172.21.15.46:6789/0 conn(0x7f755809b250 legacy=0x7f755813c830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 >> v1:172.21.15.45:6789/0 conn(0x7f755813b000 legacy=0x7f755813d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient: _finish_hunting 0 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 monclient: found mon.c 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7558135770 con 0x7f755809b8e0 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient: _finish_auth 0 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient: _check_auth_tickets 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 5 monclient: authenticate success, global_id 144296 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient: _renew_subs 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7558135df0 con 0x7f755809b8e0 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 client.144296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient: _renew_subs 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f75581d2b30 con 0x7f755809b8e0 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 librados: init done 2024-09-16T03:18:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7540001700 con 0x7f755809b8e0 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7540004c40 con 0x7f755809b8e0 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f7554d99640 4 set_mon_vals no callback set 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient: dump: 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f753c0065d0 con 0x7f753c002870 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 client.144296.objecter ms_handle_connect 0x7f753c002870 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 client.144296.objecter resend_mon_ops 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 880209232 0 0) 0x7f753c0065d0 con 0x7f753c002870 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 10 monclient(hunting): my global_id is 134397 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f755813a4a0 con 0x7f753c002870 2024-09-16T03:18:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.400+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7540010cb0 con 0x7f755809b8e0 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(539..539 src has 99..539) ==== 5874+0+0 (unknown 909202506 0 0) 0x7f754004ba90 con 0x7f755809b8e0 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 10 client.144296.objecter ms_dispatch 0x7f7558000ba0 osd_map(539..539 src has 99..539) 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 3 client.144296.objecter handle_osd_map got epochs [539,539] > 0 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 3 client.144296.objecter handle_osd_map decoding full epoch 539 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 20 client.144296.objecter dump_active .. 0 homeless 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2370107162 0 0) 0x7f755813a4a0 con 0x7f753c002870 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f753c0065d0 con 0x7f753c002870 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7548004280 con 0x7f753c002870 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 10 monclient: dump: 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f7556d9d640 10 monclient: get_auth_request con 0x7f75380401d0 auth_method 0 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2375381529 0 0) 0x7f753c0065d0 con 0x7f753c002870 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f753c00a170 con 0x7f753c002870 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 10 client.144296.objecter ms_handle_connect 0x7f75380401d0 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7548006580 con 0x7f753c002870 2024-09-16T03:18:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 >> v1:172.21.15.45:6789/0 conn(0x7f753c002870 legacy=0x7f753c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.404+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 >> v1:172.21.15.45:6789/0 conn(0x7f753c002870 legacy=0x7f753c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f755cb94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f753c009e80 con 0x7f753c009140 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 10 client.144296.objecter ms_handle_connect 0x7f753c009140 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 10 client.144296.objecter resend_mon_ops 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549647395 0 0) 0x7f753c009e80 con 0x7f753c009140 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 10 monclient(hunting): my global_id is 134400 2024-09-16T03:18:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f753c0065d0 con 0x7f753c009140 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2953331121 0 0) 0x7f753c0065d0 con 0x7f753c009140 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f753c009e80 con 0x7f753c009140 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7548005e20 con 0x7f753c009140 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 10 monclient: dump: 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3404588184 0 0) 0x7f753c009e80 con 0x7f753c009140 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.408+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f753c00c420 con 0x7f753c009140 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4293 (unknown 0 0 2393223868) 0x7f75480067b0 con 0x7f753c009140 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f754effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f754effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 >> v1:172.21.15.45:6789/0 conn(0x7f753c009140 legacy=0x7f753c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f754effd640 1 -- 172.21.15.45:0/2371329530 >> v1:172.21.15.45:6789/0 conn(0x7f753c009140 legacy=0x7f753c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:27.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 58, 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 240637, 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:18:27.428 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:18:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:18:27.430 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:18:27.431 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.432 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.434 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:27.435 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:27.436 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.437 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.438 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.439 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:18:27.439 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 10 librados: watch_flush enter 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 10 librados: watch_flush exit 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 20 client.144296.objecter shutdown clearing up homeless session... 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 10 client.144296.objecter successfully canceled tick 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 >> v1:172.21.15.46:6812/149360280 conn(0x7f75380401d0 legacy=0x7f7538042690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 10 monclient: shutdown 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 >> v1:172.21.15.45:6790/0 conn(0x7f755809b8e0 legacy=0x7f755813cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f7556d9d640 1 -- 172.21.15.45:0/2371329530 reap_dead start 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f7556d9d640 1 -- 172.21.15.45:0/2371329530 reap_dead start 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 shutdown_connections 2024-09-16T03:18:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 >> 172.21.15.45:0/2371329530 conn(0x7f75580a2bd0 msgr2=0x7f755809dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:27.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 shutdown_connections 2024-09-16T03:18:27.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 1 -- 172.21.15.45:0/2371329530 wait complete. 2024-09-16T03:18:27.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.412+0000 7f755cb94640 1 librados: shutdown 2024-09-16T03:18:27.442 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:18:27.442 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:18:27.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:18:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: get_monmap_and_config 2024-09-16T03:18:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: build_initial_monmap 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: monmap: 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:18:27.556314+0000 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:18:27.556314+0000 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:18:27.554 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 Processor -- start 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 -- start start 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: init 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient(hunting): picked mon.noname-a con 0x7f321c0a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient(hunting): picked mon.noname-b con 0x7f321c0a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient(hunting): picked mon.noname-c con 0x7f321c073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f321c09d9e0 con 0x7f321c0a4d20 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f321c09dc20 con 0x7f321c073d40 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f321c073360 con 0x7f321c0a5810 2024-09-16T03:18:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient(hunting): authenticate will time out at 1798.205637s 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f321c0a4d20 0x7f321c0a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37562/0 (socket says 172.21.15.45:37562) 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321bfff640 1 -- 172.21.15.45:0/634444564 learned_addr learned my addr 172.21.15.45:0/634444564 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f3220e8c640 1 -- 172.21.15.45:0/634444564 >> v1:172.21.15.46:6789/0 conn(0x7f321c073d40 legacy=0x7f321c140660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557181744 0 0) 0x7f321c073360 con 0x7f321c0a5810 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient(hunting): my global_id is 144299 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31fc003530 con 0x7f321c0a5810 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3585241239 0 0) 0x7f321c09d9e0 con 0x7f321c0a4d20 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient(hunting): my global_id is 134403 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f321c073360 con 0x7f321c0a4d20 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2846041408 0 0) 0x7f31fc003530 con 0x7f321c0a5810 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 >> v1:172.21.15.46:6789/0 conn(0x7f321c073d40 legacy=0x7f321c140660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 >> v1:172.21.15.45:6789/0 conn(0x7f321c0a4d20 legacy=0x7f321c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: _finish_hunting 0 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 monclient: found mon.noname-b 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321c09cf30 con 0x7f321c0a5810 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: _finish_auth 0 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: _check_auth_tickets 2024-09-16T03:18:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f32040016a0 con 0x7f321c0a5810 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 5 monclient: authenticate success, global_id 144299 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: dump: 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3204001840 con 0x7f321c0a5810 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 1 -- 172.21.15.45:0/634444564 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3204001cd0 con 0x7f321c0a5810 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f321affd640 10 monclient: dump: 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: get_monmap_and_config success 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 4 set_mon_vals no callback set 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: shutdown 2024-09-16T03:18:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 -- 172.21.15.45:0/634444564 >> v1:172.21.15.45:6790/0 conn(0x7f321c0a5810 legacy=0x7f321c13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 -- 172.21.15.45:0/634444564 shutdown_connections 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 -- 172.21.15.45:0/634444564 >> 172.21.15.45:0/634444564 conn(0x7f321c0aa720 msgr2=0x7f321c0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 -- 172.21.15.45:0/634444564 shutdown_connections 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 1 -- 172.21.15.45:0/634444564 wait complete. 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: build_initial_monmap 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.552+0000 7f32228ef640 10 monclient: monmap: 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:18:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 librados: starting msgr at 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 librados: starting objecter 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 Processor -- start 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 -- start start 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 librados: setting wanted keys 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 librados: calling monclient init 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient: init 2024-09-16T03:18:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient: _reopen_session rank -1 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient(hunting): picked mon.noname-c con 0x7f321c073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient(hunting): picked mon.noname-b con 0x7f321c0a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient(hunting): picked mon.noname-a con 0x7f321c0a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f321c136e40 con 0x7f321c0a5810 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f321c134360 con 0x7f321c0a4d20 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f321c0a1fe0 con 0x7f321c073d40 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient(hunting): _renew_subs 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient(hunting): authenticate will time out at 1798.208806s 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f321bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f321c073d40 0x7f321c133540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:54662/0 (socket says 172.21.15.45:54662) 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f321bfff640 1 -- 172.21.15.45:0/118426171 learned_addr learned my addr 172.21.15.45:0/118426171 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f321b7fe640 1 -- 172.21.15.45:0/118426171 >> v1:172.21.15.46:6789/0 conn(0x7f321c0a4d20 legacy=0x7f321c133c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 client.?.objecter ms_handle_connect 0x7f321c073d40 2024-09-16T03:18:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 client.?.objecter ms_handle_connect 0x7f321c0a5810 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1803104167 0 0) 0x7f321c0a1fe0 con 0x7f321c073d40 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): my global_id is 144302 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3200004ec0 con 0x7f321c073d40 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 880554568 0 0) 0x7f321c136e40 con 0x7f321c0a5810 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): my global_id is 134406 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f321c0a1fe0 con 0x7f321c0a5810 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1696359460 0 0) 0x7f3200004ec0 con 0x7f321c073d40 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f321c136e40 con 0x7f321c073d40 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3214001740 con 0x7f321c073d40 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): dump: 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1483353055 0 0) 0x7f321c0a1fe0 con 0x7f321c0a5810 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3200004ec0 con 0x7f321c0a5810 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f320c0040c0 con 0x7f321c0a5810 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): dump: 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 796097932 0 0) 0x7f321c136e40 con 0x7f321c073d40 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 >> v1:172.21.15.46:6789/0 conn(0x7f321c0a4d20 legacy=0x7f321c133c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 >> v1:172.21.15.45:6789/0 conn(0x7f321c0a5810 legacy=0x7f321c1d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:18:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 monclient: found mon.c 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321c0a1a90 con 0x7f321c073d40 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: _finish_auth 0 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 5 monclient: authenticate success, global_id 144302 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient: _renew_subs 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f321c0a1510 con 0x7f321c073d40 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 client.144302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient: _renew_subs 2024-09-16T03:18:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:18:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f321c137540 con 0x7f321c073d40 2024-09-16T03:18:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 librados: init done 2024-09-16T03:18:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3214001980 con 0x7f321c073d40 2024-09-16T03:18:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:18:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3214004fc0 con 0x7f321c073d40 2024-09-16T03:18:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f321a7fc640 4 set_mon_vals no callback set 2024-09-16T03:18:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: dump: 2024-09-16T03:18:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f31f40065d0 con 0x7f31f4002870 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f3214011030 con 0x7f321c073d40 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 client.144302.objecter ms_handle_connect 0x7f31f4002870 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 client.144302.objecter resend_mon_ops 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(539..539 src has 99..539) ==== 5874+0+0 (unknown 909202506 0 0) 0x7f321404bee0 con 0x7f321c073d40 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 client.144302.objecter ms_dispatch 0x7f321c000ba0 osd_map(539..539 src has 99..539) 2024-09-16T03:18:27.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 3 client.144302.objecter handle_osd_map got epochs [539,539] > 0 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 3 client.144302.objecter handle_osd_map decoding full epoch 539 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 20 client.144302.objecter dump_active .. 0 homeless 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3120267659 0 0) 0x7f31f40065d0 con 0x7f31f4002870 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient(hunting): my global_id is 144305 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f321c136e40 con 0x7f31f4002870 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3283170930 0 0) 0x7f321c136e40 con 0x7f31f4002870 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3220e8c640 10 monclient: get_auth_request con 0x7f320003df80 auth_method 0 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31f40065d0 con 0x7f31f4002870 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3204001910 con 0x7f31f4002870 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: dump: 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 855762149 0 0) 0x7f31f40065d0 con 0x7f31f4002870 2024-09-16T03:18:27.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31f400a170 con 0x7f31f4002870 2024-09-16T03:18:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 client.144302.objecter ms_handle_connect 0x7f320003df80 2024-09-16T03:18:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3204001b30 con 0x7f31f4002870 2024-09-16T03:18:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:18:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:18:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 >> v1:172.21.15.45:6790/0 conn(0x7f31f4002870 legacy=0x7f31f4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.556+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 >> v1:172.21.15.45:6790/0 conn(0x7f31f4002870 legacy=0x7f31f4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:18:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f31f4009e60 con 0x7f31f4009120 2024-09-16T03:18:27.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 client.144302.objecter ms_handle_connect 0x7f31f4009120 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 client.144302.objecter resend_mon_ops 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310574169 0 0) 0x7f31f4009e60 con 0x7f31f4009120 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 monclient(hunting): my global_id is 144308 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31f40065d0 con 0x7f31f4009120 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2393438942 0 0) 0x7f31f40065d0 con 0x7f31f4009120 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31f4009e60 con 0x7f31f4009120 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f32040049e0 con 0x7f31f4009120 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:18:27.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 monclient: dump: 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4059714766 0 0) 0x7f31f4009e60 con 0x7f31f4009120 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f31f400c400 con 0x7f31f4009120 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4124 (unknown 0 0 3099931695) 0x7f3204005e50 con 0x7f31f4009120 2024-09-16T03:18:27.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:18:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:18:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 >> v1:172.21.15.45:6790/0 conn(0x7f31f4009120 legacy=0x7f31f40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3218ff9640 1 -- 172.21.15.45:0/118426171 >> v1:172.21.15.45:6790/0 conn(0x7f31f4009120 legacy=0x7f31f40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:18:27.585 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:18:27.585 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:27.585 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:27.585 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:18:27.585 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 58, 2024-09-16T03:18:27.585 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 132638, 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:27.586 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:27.587 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:18:27.588 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:18:27.589 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:18:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:27.591 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.593 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.594 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.595 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:18:27.596 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 10 librados: watch_flush enter 2024-09-16T03:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 10 librados: watch_flush exit 2024-09-16T03:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 20 client.144302.objecter shutdown clearing up homeless session... 2024-09-16T03:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 10 client.144302.objecter successfully canceled tick 2024-09-16T03:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 >> v1:172.21.15.46:6812/149360280 conn(0x7f320003df80 legacy=0x7f3200040440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 10 monclient: shutdown 2024-09-16T03:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:18:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 >> v1:172.21.15.45:6790/0 conn(0x7f321c073d40 legacy=0x7f321c133540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:18:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3220e8c640 1 -- 172.21.15.45:0/118426171 reap_dead start 2024-09-16T03:18:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f3220e8c640 1 -- 172.21.15.45:0/118426171 reap_dead start 2024-09-16T03:18:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 shutdown_connections 2024-09-16T03:18:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 >> 172.21.15.45:0/118426171 conn(0x7f321c0aa720 msgr2=0x7f321c0a8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:18:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 shutdown_connections 2024-09-16T03:18:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 1 -- 172.21.15.45:0/118426171 wait complete. 2024-09-16T03:18:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:18:27.564+0000 7f32228ef640 1 librados: shutdown 2024-09-16T03:18:27.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:18:27.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:19:08.925 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:19:08.925 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9003 ms) 2024-09-16T03:19:08.925 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6021 ms) 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5050 ms) 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25895 ms) 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14012 ms) 2024-09-16T03:19:08.926 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12031 ms) 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: 00000012 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (26961 ms) 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6068 ms) 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (627624 ms total) 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-16T03:19:08.927 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1033 ms) 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10993 ms) 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17994 ms) 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (4 ms) 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30024 ms total) 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6993 ms) 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8042 ms) 2024-09-16T03:19:08.928 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: waiting for scrub... 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: done waiting 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (25150 ms) 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (29939 ms) 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7893 ms) 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8065 ms) 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-16T03:19:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9911 ms) 2024-09-16T03:19:08.930 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-16T03:19:57.600 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:19:57.601 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:19:57.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:19:57.605 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:19:57.605 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:19:57.605 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:19:57.605 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:19:57.709 INFO:tasks.ceph.mon.b.smithi046.stderr:2024-09-16T03:19:57.706+0000 7fe3c9234e00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 571 -> 0 (sync_force? mon rebuild?) 2024-09-16T03:20:12.606 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:20:12.606 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:20:12.606 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:20:12.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:20:12.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient: get_monmap_and_config 2024-09-16T03:20:12.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient: build_initial_monmap 2024-09-16T03:20:12.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient: monmap: 2024-09-16T03:20:12.744 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:12.744 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:12.744 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:12.746369+0000 2024-09-16T03:20:12.745 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:12.746369+0000 2024-09-16T03:20:12.745 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:12.745 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:12.745 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:12.745 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:12.745 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:12.745 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:12.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 1 Processor -- start 2024-09-16T03:20:12.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 1 -- start start 2024-09-16T03:20:12.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient: init 2024-09-16T03:20:12.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff990135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff99009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff99013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff99009f7e0 con 0x7ff990135950 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff99009b160 con 0x7ff99013c010 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff99009b440 con 0x7ff99009fc40 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.742+0000 7ff9984c6640 10 monclient(hunting): authenticate will time out at 1903.395883s 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff996262640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff990135950 0x7ff990135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37804/0 (socket says 172.21.15.45:37804) 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff996262640 1 -- 172.21.15.45:0/3759841975 learned_addr learned my addr 172.21.15.45:0/3759841975 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995a61640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff99009fc40 0x7ff99013a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43224/0 (socket says 172.21.15.45:43224) 2024-09-16T03:20:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506347823 0 0) 0x7ff99009f7e0 con 0x7ff990135950 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient(hunting): _negotiate 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient(hunting): my global_id is 134409 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff97c003530 con 0x7ff990135950 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793972709 0 0) 0x7ff99009b440 con 0x7ff99009fc40 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient(hunting): _negotiate 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient(hunting): my global_id is 144311 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff99009f7e0 con 0x7ff99009fc40 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1172738343 0 0) 0x7ff99009f7e0 con 0x7ff99009fc40 2024-09-16T03:20:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 >> v1:172.21.15.46:6789/0 conn(0x7ff99013c010 legacy=0x7ff99013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 >> v1:172.21.15.45:6789/0 conn(0x7ff990135950 legacy=0x7ff990135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: _finish_hunting 0 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 monclient: found mon.noname-b 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff990136440 con 0x7ff99009fc40 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: _finish_auth 0 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: _check_auth_tickets 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff980001680 con 0x7ff99009fc40 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 5 monclient: authenticate success, global_id 144311 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:12.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: dump: 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:12.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9800018a0 con 0x7ff99009fc40 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 1 -- 172.21.15.45:0/3759841975 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff980001cb0 con 0x7ff99009fc40 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 4 set_mon_vals no callback set 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995260640 10 monclient: dump: 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:12.757 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient: shutdown 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 -- 172.21.15.45:0/3759841975 >> v1:172.21.15.45:6790/0 conn(0x7ff99009fc40 legacy=0x7ff99013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 -- 172.21.15.45:0/3759841975 shutdown_connections 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 -- 172.21.15.45:0/3759841975 >> 172.21.15.45:0/3759841975 conn(0x7ff9900a2bd0 msgr2=0x7ff9900a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 -- 172.21.15.45:0/3759841975 shutdown_connections 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 -- 172.21.15.45:0/3759841975 wait complete. 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient: build_initial_monmap 2024-09-16T03:20:12.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient: monmap: 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 librados: starting msgr at 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 librados: starting objecter 2024-09-16T03:20:12.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 Processor -- start 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 -- start start 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 librados: setting wanted keys 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 librados: calling monclient init 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient: init 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff99009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff990135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff99013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff99009ebf0 con 0x7ff990135950 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9900735e0 con 0x7ff99013c010 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff99013bb40 con 0x7ff99009fc40 2024-09-16T03:20:12.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff9984c6640 10 monclient(hunting): authenticate will time out at 1903.399262s 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995a61640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff990135950 0x7ff990072ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37814/0 (socket says 172.21.15.45:37814) 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff996262640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff99009fc40 0x7ff9900727c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43228/0 (socket says 172.21.15.45:43228) 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff995a61640 1 -- 172.21.15.45:0/4031777045 learned_addr learned my addr 172.21.15.45:0/4031777045 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 client.?.objecter ms_handle_connect 0x7ff990135950 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 client.?.objecter ms_handle_connect 0x7ff99009fc40 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 53607698 0 0) 0x7ff99013bb40 con 0x7ff99009fc40 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:12.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): my global_id is 144314 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff974004ec0 con 0x7ff99009fc40 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1874026986 0 0) 0x7ff99009ebf0 con 0x7ff990135950 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): my global_id is 134412 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff99013bb40 con 0x7ff990135950 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 client.?.objecter ms_handle_connect 0x7ff99013c010 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1282147225 0 0) 0x7ff974004ec0 con 0x7ff99009fc40 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff99009ebf0 con 0x7ff99009fc40 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff98c0012e0 con 0x7ff99009fc40 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:12.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): dump: 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:12.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 515276007 0 0) 0x7ff99013bb40 con 0x7ff990135950 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.746+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff974004ec0 con 0x7ff990135950 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff980001cb0 con 0x7ff990135950 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient(hunting): dump: 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1641250269 0 0) 0x7ff9900735e0 con 0x7ff99013c010 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient(hunting): my global_id is 164101 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff99013bb40 con 0x7ff99013c010 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2485511117 0 0) 0x7ff99009ebf0 con 0x7ff99009fc40 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 >> v1:172.21.15.46:6789/0 conn(0x7ff99013c010 legacy=0x7ff9901d4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 >> v1:172.21.15.45:6789/0 conn(0x7ff990135950 legacy=0x7ff990072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 monclient: found mon.c 2024-09-16T03:20:12.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99013b6f0 con 0x7ff99009fc40 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient: _finish_auth 0 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff9984c6640 5 monclient: authenticate success, global_id 144314 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff9984c6640 10 monclient: _renew_subs 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff9984c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff9984c6640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff99009d2c0 con 0x7ff99009fc40 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff9984c6640 10 client.144314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff9984c6640 10 monclient: _renew_subs 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff9984c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff9984c6640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff99013af20 con 0x7ff99009fc40 2024-09-16T03:20:12.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff9984c6640 1 librados: init done 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff98c0047f0 con 0x7ff99009fc40 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff98c004b40 con 0x7ff99009fc40 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 monclient: dump: 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:12.767 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff987fff640 4 set_mon_vals no callback set 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff984ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7ff98c010bf0 con 0x7ff99009fc40 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff984ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff984ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff984ff9640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff96c005090 con 0x7ff99009fc40 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(627..627 src has 99..627) ==== 6012+0+0 (unknown 1002724289 0 0) 0x7ff98c04b990 con 0x7ff99009fc40 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 client.144314.objecter ms_dispatch 0x7ff990000ba0 osd_map(627..627 src has 99..627) 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 3 client.144314.objecter handle_osd_map got epochs [627,627] > 0 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 3 client.144314.objecter handle_osd_map decoding full epoch 627 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 20 client.144314.objecter dump_active .. 0 homeless 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff995a61640 10 monclient: get_auth_request con 0x7ff97403e210 auth_method 0 2024-09-16T03:20:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.750+0000 7ff986ffd640 10 client.144314.objecter ms_handle_connect 0x7ff97403e210 2024-09-16T03:20:12.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.754+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7ff98c04de20 con 0x7ff99009fc40 2024-09-16T03:20:12.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.754+0000 7ff986ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:12.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:12.754+0000 7ff986ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.002+0000 7ff984ff9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:20:13.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.002+0000 7ff984ff9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:13.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.002+0000 7ff984ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.002+0000 7ff984ff9640 1 -- 172.21.15.45:0/4031777045 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff96c005380 con 0x7ff99009fc40 2024-09-16T03:20:13.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.002+0000 7ff986ffd640 1 -- 172.21.15.45:0/4031777045 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2548645196) 0x7ff98c04b590 con 0x7ff99009fc40 2024-09-16T03:20:13.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.002+0000 7ff986ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:13.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.002+0000 7ff986ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:13.005 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:20:13.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 10 librados: watch_flush enter 2024-09-16T03:20:13.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 10 librados: watch_flush exit 2024-09-16T03:20:13.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 20 client.144314.objecter shutdown clearing up homeless session... 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 10 client.144314.objecter successfully canceled tick 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 1 -- 172.21.15.45:0/4031777045 >> v1:172.21.15.46:6812/149360280 conn(0x7ff97403e210 legacy=0x7ff9740406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 10 monclient: shutdown 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 1 -- 172.21.15.45:0/4031777045 >> v1:172.21.15.45:6790/0 conn(0x7ff99009fc40 legacy=0x7ff9900727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 1 -- 172.21.15.45:0/4031777045 shutdown_connections 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 1 -- 172.21.15.45:0/4031777045 >> 172.21.15.45:0/4031777045 conn(0x7ff9900a2bd0 msgr2=0x7ff99013ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 1 -- 172.21.15.45:0/4031777045 shutdown_connections 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 1 -- 172.21.15.45:0/4031777045 wait complete. 2024-09-16T03:20:13.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.006+0000 7ff984ff9640 1 librados: shutdown 2024-09-16T03:20:13.024 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:20:13.024 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:20:13.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:20:13.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient: get_monmap_and_config 2024-09-16T03:20:13.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient: build_initial_monmap 2024-09-16T03:20:13.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient: monmap: 2024-09-16T03:20:13.147 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:13.147 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.148 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:13.149632+0000 2024-09-16T03:20:13.148 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:13.149632+0000 2024-09-16T03:20:13.148 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:13.148 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.148 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:13.148 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:13.148 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:13.148 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 1 Processor -- start 2024-09-16T03:20:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 1 -- start start 2024-09-16T03:20:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient: init 2024-09-16T03:20:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:20:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient(hunting): picked mon.noname-a con 0x7f4df809b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient(hunting): picked mon.noname-b con 0x7f4df809b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient(hunting): picked mon.noname-c con 0x7f4df813b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df809cc20 con 0x7f4df809b8e0 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df809d820 con 0x7f4df813b000 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df809b6a0 con 0x7f4df809b250 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 10 monclient(hunting): authenticate will time out at 1903.799120s 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4df809b8e0 0x7f4df809bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37830/0 (socket says 172.21.15.45:37830) 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df7fff640 1 -- 172.21.15.45:0/496357535 learned_addr learned my addr 172.21.15.45:0/496357535 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df77fe640 1 --1- 172.21.15.45:0/496357535 >> v1:172.21.15.45:6790/0 conn(0x7f4df809b250 0x7f4df81398e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43238/0 (socket says 172.21.15.45:43238) 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586410774 0 0) 0x7f4df809b6a0 con 0x7f4df809b250 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient(hunting): my global_id is 144317 2024-09-16T03:20:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4de4003530 con 0x7f4df809b250 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2291853000 0 0) 0x7f4df809cc20 con 0x7f4df809b8e0 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient(hunting): my global_id is 134415 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df809b6a0 con 0x7f4df809b8e0 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2921430214 0 0) 0x7f4de4003530 con 0x7f4df809b250 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 >> v1:172.21.15.46:6789/0 conn(0x7f4df813b000 legacy=0x7f4df813d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 >> v1:172.21.15.45:6789/0 conn(0x7f4df809b8e0 legacy=0x7f4df809bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 monclient: found mon.noname-b 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df80a21f0 con 0x7f4df809b250 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4de00016a0 con 0x7f4df809b250 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 5 monclient: authenticate success, global_id 144317 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.146+0000 7f4dfe717640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df6ffd640 10 monclient: dump: 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4de00018c0 con 0x7f4df809b250 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df6ffd640 1 -- 172.21.15.45:0/496357535 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4de0001cd0 con 0x7f4df809b250 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 4 set_mon_vals no callback set 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df6ffd640 10 monclient: dump: 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: shutdown 2024-09-16T03:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- 172.21.15.45:0/496357535 >> v1:172.21.15.45:6790/0 conn(0x7f4df809b250 legacy=0x7f4df81398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- 172.21.15.45:0/496357535 shutdown_connections 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- 172.21.15.45:0/496357535 >> 172.21.15.45:0/496357535 conn(0x7f4df80a2bd0 msgr2=0x7f4df80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- 172.21.15.45:0/496357535 shutdown_connections 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- 172.21.15.45:0/496357535 wait complete. 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: build_initial_monmap 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: monmap: 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 librados: starting msgr at 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 librados: starting objecter 2024-09-16T03:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 Processor -- start 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- start start 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 librados: setting wanted keys 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 librados: calling monclient init 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: init 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient(hunting): picked mon.noname-a con 0x7f4df809b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient(hunting): picked mon.noname-b con 0x7f4df809b8e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient(hunting): picked mon.noname-c con 0x7f4df813b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df8092280 con 0x7f4df809b250 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df808fdf0 con 0x7f4df809b8e0 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df813d780 con 0x7f4df813b000 2024-09-16T03:20:13.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient(hunting): authenticate will time out at 1903.802216s 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfccb4640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4df813b000 0x7f4df81d2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43250/0 (socket says 172.21.15.45:43250) 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4df809b250 0x7f4df808efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37840/0 (socket says 172.21.15.45:37840) 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfccb4640 1 -- 172.21.15.45:0/1912960214 learned_addr learned my addr 172.21.15.45:0/1912960214 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 client.?.objecter ms_handle_connect 0x7f4df813b000 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 client.?.objecter ms_handle_connect 0x7f4df809b250 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169430621 0 0) 0x7f4df8092280 con 0x7f4df809b250 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): my global_id is 134418 2024-09-16T03:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dd8004ec0 con 0x7f4df809b250 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 client.?.objecter ms_handle_connect 0x7f4df809b8e0 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243346960 0 0) 0x7f4df813d780 con 0x7f4df813b000 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): my global_id is 144320 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df8092280 con 0x7f4df813b000 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 65266478 0 0) 0x7f4dd8004ec0 con 0x7f4df809b250 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4df813d780 con 0x7f4df809b250 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4de80012e0 con 0x7f4df809b250 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): dump: 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1976183713 0 0) 0x7f4df8092280 con 0x7f4df813b000 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4dd8004ec0 con 0x7f4df813b000 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3592274287 0 0) 0x7f4df808fdf0 con 0x7f4df809b8e0 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): my global_id is 164107 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df8092280 con 0x7f4df809b8e0 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4df0001c40 con 0x7f4df813b000 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient(hunting): dump: 2024-09-16T03:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1571499242 0 0) 0x7f4df813d780 con 0x7f4df809b250 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 >> v1:172.21.15.45:6790/0 conn(0x7f4df813b000 legacy=0x7f4df81d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 >> v1:172.21.15.46:6789/0 conn(0x7f4df809b8e0 legacy=0x7f4df808f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 monclient: found mon.a 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df813dc90 con 0x7f4df809b250 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient: _finish_auth 0 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 5 monclient: authenticate success, global_id 134418 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: _renew_subs 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4df809d2c0 con 0x7f4df809b250 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 client.134418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: _renew_subs 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4df8091030 con 0x7f4df809b250 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4dfe717640 1 librados: init done 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4de80047f0 con 0x7f4df809b250 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4de8004b40 con 0x7f4df809b250 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df67fc640 4 set_mon_vals no callback set 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.150+0000 7f4df4ff9640 10 monclient: dump: 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4dfe717640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4ddc0065d0 con 0x7f4ddc002870 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 client.134418.objecter ms_handle_connect 0x7f4ddc002870 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 client.134418.objecter resend_mon_ops 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4de8010bb0 con 0x7f4df809b250 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(628..628 src has 99..628) ==== 6004+0+0 (unknown 2367076389 0 0) 0x7f4de804b8d0 con 0x7f4df809b250 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 client.134418.objecter ms_dispatch 0x7f4df8000ba0 osd_map(628..628 src has 99..628) 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 3 client.134418.objecter handle_osd_map got epochs [628,628] > 0 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 3 client.134418.objecter handle_osd_map decoding full epoch 628 2024-09-16T03:20:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 20 client.134418.objecter dump_active .. 0 homeless 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107249568 0 0) 0x7f4ddc0065d0 con 0x7f4ddc002870 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 monclient(hunting): my global_id is 134421 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df813d780 con 0x7f4ddc002870 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1978604440 0 0) 0x7f4df813d780 con 0x7f4ddc002870 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4dfccb4640 10 monclient: get_auth_request con 0x7f4dd803e240 auth_method 0 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ddc0065d0 con 0x7f4ddc002870 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4de0001720 con 0x7f4ddc002870 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 monclient: dump: 2024-09-16T03:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 119719303 0 0) 0x7f4ddc0065d0 con 0x7f4ddc002870 2024-09-16T03:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ddc00a170 con 0x7f4ddc002870 2024-09-16T03:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 client.134418.objecter ms_handle_connect 0x7f4dd803e240 2024-09-16T03:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4de0001a80 con 0x7f4ddc002870 2024-09-16T03:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 >> v1:172.21.15.45:6789/0 conn(0x7f4ddc002870 legacy=0x7f4ddc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.154+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 >> v1:172.21.15.45:6789/0 conn(0x7f4ddc002870 legacy=0x7f4ddc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:13.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.158+0000 7f4dfe717640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.158+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4ddc009e80 con 0x7f4ddc009140 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.158+0000 7f4df4ff9640 10 client.134418.objecter ms_handle_connect 0x7f4ddc009140 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.158+0000 7f4df4ff9640 10 client.134418.objecter resend_mon_ops 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2480607735 0 0) 0x7f4ddc009e80 con 0x7f4ddc009140 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 10 monclient(hunting): my global_id is 134424 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ddc0065d0 con 0x7f4ddc009140 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2345010519 0 0) 0x7f4ddc0065d0 con 0x7f4ddc009140 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ddc009e80 con 0x7f4ddc009140 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4de0004a30 con 0x7f4ddc009140 2024-09-16T03:20:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 10 monclient: dump: 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1339595968 0 0) 0x7f4ddc009e80 con 0x7f4ddc009140 2024-09-16T03:20:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ddc00c420 con 0x7f4ddc009140 2024-09-16T03:20:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4304 (unknown 0 0 2797279901) 0x7f4de0005ea0 con 0x7f4ddc009140 2024-09-16T03:20:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:20:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 >> v1:172.21.15.45:6789/0 conn(0x7f4ddc009140 legacy=0x7f4ddc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4df4ff9640 1 -- 172.21.15.45:0/1912960214 >> v1:172.21.15.45:6789/0 conn(0x7f4ddc009140 legacy=0x7f4ddc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:13.179 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:20:13.179 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:13.179 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:13.179 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 62, 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 346391, 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:20:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:20:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:20:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:20:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:20:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:20:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:20:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:20:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:20:13.196 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:20:13.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 10 librados: watch_flush enter 2024-09-16T03:20:13.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 10 librados: watch_flush exit 2024-09-16T03:20:13.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 20 client.134418.objecter shutdown clearing up homeless session... 2024-09-16T03:20:13.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 10 client.134418.objecter successfully canceled tick 2024-09-16T03:20:13.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 >> v1:172.21.15.46:6812/149360280 conn(0x7f4dd803e240 legacy=0x7f4dd8040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 10 monclient: shutdown 2024-09-16T03:20:13.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:13.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 >> v1:172.21.15.45:6789/0 conn(0x7f4df809b250 legacy=0x7f4df808efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfccb4640 1 -- 172.21.15.45:0/1912960214 reap_dead start 2024-09-16T03:20:13.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfccb4640 1 -- 172.21.15.45:0/1912960214 reap_dead start 2024-09-16T03:20:13.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 shutdown_connections 2024-09-16T03:20:13.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 >> 172.21.15.45:0/1912960214 conn(0x7f4df80a2bd0 msgr2=0x7f4df809efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:13.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 shutdown_connections 2024-09-16T03:20:13.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 1 -- 172.21.15.45:0/1912960214 wait complete. 2024-09-16T03:20:13.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.162+0000 7f4dfe717640 1 librados: shutdown 2024-09-16T03:20:13.201 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:20:13.201 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:20:13.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:20:13.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient: get_monmap_and_config 2024-09-16T03:20:13.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient: build_initial_monmap 2024-09-16T03:20:13.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient: monmap: 2024-09-16T03:20:13.319 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:13.319 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.319 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:13.321180+0000 2024-09-16T03:20:13.320 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:13.321180+0000 2024-09-16T03:20:13.320 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:13.320 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.320 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:13.320 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:13.320 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:13.320 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 1 Processor -- start 2024-09-16T03:20:13.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 1 -- start start 2024-09-16T03:20:13.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient: init 2024-09-16T03:20:13.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:13.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:20:13.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient(hunting): picked mon.noname-a con 0x7f94f4135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:13.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient(hunting): picked mon.noname-b con 0x7f94f409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:13.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient(hunting): picked mon.noname-c con 0x7f94f413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f409f7e0 con 0x7f94f4135950 2024-09-16T03:20:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f409b160 con 0x7f94f413c010 2024-09-16T03:20:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f409b440 con 0x7f94f409fc40 2024-09-16T03:20:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient(hunting): authenticate will time out at 1903.970466s 2024-09-16T03:20:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f3fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f94f4135950 0x7f94f4135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37880/0 (socket says 172.21.15.45:37880) 2024-09-16T03:20:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f3fff640 1 -- 172.21.15.45:0/1337744596 learned_addr learned my addr 172.21.15.45:0/1337744596 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 253855656 0 0) 0x7f94f409b440 con 0x7f94f409fc40 2024-09-16T03:20:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): my global_id is 144323 2024-09-16T03:20:13.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94dc003530 con 0x7f94f409fc40 2024-09-16T03:20:13.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137734158 0 0) 0x7f94f409f7e0 con 0x7f94f4135950 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): my global_id is 134427 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f409b440 con 0x7f94f4135950 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3403598273 0 0) 0x7f94f409b160 con 0x7f94f413c010 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient(hunting): my global_id is 164110 2024-09-16T03:20:13.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f409f7e0 con 0x7f94f413c010 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2233092847 0 0) 0x7f94dc003530 con 0x7f94f409fc40 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 >> v1:172.21.15.46:6789/0 conn(0x7f94f413c010 legacy=0x7f94f413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 >> v1:172.21.15.45:6789/0 conn(0x7f94f4135950 legacy=0x7f94f4135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 monclient: found mon.noname-b 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f4136440 con 0x7f94f409fc40 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f94e40016a0 con 0x7f94f409fc40 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 5 monclient: authenticate success, global_id 144323 2024-09-16T03:20:13.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: dump: 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.328 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f94e4001840 con 0x7f94f409fc40 2024-09-16T03:20:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 1 -- 172.21.15.45:0/1337744596 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f94e4001cd0 con 0x7f94f409fc40 2024-09-16T03:20:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94fa812640 4 set_mon_vals no callback set 2024-09-16T03:20:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.318+0000 7f94f2ffd640 10 monclient: dump: 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: shutdown 2024-09-16T03:20:13.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- 172.21.15.45:0/1337744596 >> v1:172.21.15.45:6790/0 conn(0x7f94f409fc40 legacy=0x7f94f413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- 172.21.15.45:0/1337744596 shutdown_connections 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- 172.21.15.45:0/1337744596 >> 172.21.15.45:0/1337744596 conn(0x7f94f40a2bd0 msgr2=0x7f94f40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- 172.21.15.45:0/1337744596 shutdown_connections 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- 172.21.15.45:0/1337744596 wait complete. 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: build_initial_monmap 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: monmap: 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:13.331 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 librados: starting msgr at 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 librados: starting objecter 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 Processor -- start 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- start start 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 librados: setting wanted keys 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 librados: calling monclient init 2024-09-16T03:20:13.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: init 2024-09-16T03:20:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:20:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient(hunting): picked mon.noname-c con 0x7f94f409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient(hunting): picked mon.noname-a con 0x7f94f4135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient(hunting): picked mon.noname-b con 0x7f94f413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f413b6c0 con 0x7f94f4135950 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f409e760 con 0x7f94f413c010 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f4073db0 con 0x7f94f409fc40 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient(hunting): authenticate will time out at 1903.974008s 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f37fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f94f4135950 0x7f94f4072ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37884/0 (socket says 172.21.15.45:37884) 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f3fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f94f409fc40 0x7f94f40727c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43274/0 (socket says 172.21.15.45:43274) 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f37fe640 1 -- 172.21.15.45:0/4121432017 learned_addr learned my addr 172.21.15.45:0/4121432017 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 client.?.objecter ms_handle_connect 0x7f94f4135950 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 client.?.objecter ms_handle_connect 0x7f94f409fc40 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2231503386 0 0) 0x7f94f4136440 con 0x7f94f4135950 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): my global_id is 134430 2024-09-16T03:20:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94e0004ef0 con 0x7f94f4135950 2024-09-16T03:20:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1169311228 0 0) 0x7f94f4073db0 con 0x7f94f409fc40 2024-09-16T03:20:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): my global_id is 144326 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f4136440 con 0x7f94f409fc40 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 client.?.objecter ms_handle_connect 0x7f94f413c010 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2009054966 0 0) 0x7f94f4136440 con 0x7f94f409fc40 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94f4073db0 con 0x7f94f409fc40 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f94e80011d0 con 0x7f94f409fc40 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): dump: 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.342 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1773119328 0 0) 0x7f94e0004ef0 con 0x7f94f4135950 2024-09-16T03:20:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94f4136440 con 0x7f94f4135950 2024-09-16T03:20:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f94e4001bd0 con 0x7f94f4135950 2024-09-16T03:20:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): dump: 2024-09-16T03:20:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.345 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889347361 0 0) 0x7f94f409e760 con 0x7f94f413c010 2024-09-16T03:20:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient(hunting): my global_id is 164113 2024-09-16T03:20:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94e0004ef0 con 0x7f94f413c010 2024-09-16T03:20:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 472185801 0 0) 0x7f94f4073db0 con 0x7f94f409fc40 2024-09-16T03:20:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 >> v1:172.21.15.46:6789/0 conn(0x7f94f413c010 legacy=0x7f94f41d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 >> v1:172.21.15.45:6789/0 conn(0x7f94f4135950 legacy=0x7f94f4072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 monclient: found mon.c 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f40735e0 con 0x7f94f409fc40 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient: _finish_auth 0 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 5 monclient: authenticate success, global_id 144326 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: _renew_subs 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f94f409ed20 con 0x7f94f409fc40 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 client.144326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: _renew_subs 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f94f409f4a0 con 0x7f94f409fc40 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94fa812640 1 librados: init done 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f94e80047f0 con 0x7f94f409fc40 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f94e8004b40 con 0x7f94f409fc40 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f1ffb640 4 set_mon_vals no callback set 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 10 monclient: dump: 2024-09-16T03:20:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.322+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f94e8010bb0 con 0x7f94f409fc40 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94fa812640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f94d40065d0 con 0x7f94d4002870 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 client.144326.objecter ms_handle_connect 0x7f94d4002870 2024-09-16T03:20:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 client.144326.objecter resend_mon_ops 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(628..628 src has 99..628) ==== 6004+0+0 (unknown 2367076389 0 0) 0x7f94e804b8d0 con 0x7f94f409fc40 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 client.144326.objecter ms_dispatch 0x7f94f4000ba0 osd_map(628..628 src has 99..628) 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 3 client.144326.objecter handle_osd_map got epochs [628,628] > 0 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 3 client.144326.objecter handle_osd_map decoding full epoch 628 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 20 client.144326.objecter dump_active .. 0 homeless 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4087979222 0 0) 0x7f94d40065d0 con 0x7f94d4002870 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 monclient(hunting): my global_id is 144329 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f4073db0 con 0x7f94d4002870 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2740411728 0 0) 0x7f94f4073db0 con 0x7f94d4002870 2024-09-16T03:20:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94d40065d0 con 0x7f94d4002870 2024-09-16T03:20:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f94e40037a0 con 0x7f94d4002870 2024-09-16T03:20:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 monclient: dump: 2024-09-16T03:20:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f8daf640 10 monclient: get_auth_request con 0x7f94e003e310 auth_method 0 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3774183743 0 0) 0x7f94d40065d0 con 0x7f94d4002870 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94d400a170 con 0x7f94d4002870 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 client.144326.objecter ms_handle_connect 0x7f94e003e310 2024-09-16T03:20:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f94e4005790 con 0x7f94d4002870 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 >> v1:172.21.15.45:6790/0 conn(0x7f94d4002870 legacy=0x7f94d4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.326+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 >> v1:172.21.15.45:6790/0 conn(0x7f94d4002870 legacy=0x7f94d4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94fa812640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f94d4009e80 con 0x7f94d4009140 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 10 client.144326.objecter ms_handle_connect 0x7f94d4009140 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 10 client.144326.objecter resend_mon_ops 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857549656 0 0) 0x7f94d4009e80 con 0x7f94d4009140 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 10 monclient(hunting): my global_id is 144332 2024-09-16T03:20:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94d40065d0 con 0x7f94d4009140 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1282167467 0 0) 0x7f94d40065d0 con 0x7f94d4009140 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94d4009e80 con 0x7f94d4009140 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f94e4003de0 con 0x7f94d4009140 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.330+0000 7f94f0ff9640 10 monclient: dump: 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2235786367 0 0) 0x7f94d4009e80 con 0x7f94d4009140 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f94d400c400 con 0x7f94d4009140 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 2659515611) 0x7f94e4005940 con 0x7f94d4009140 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 >> v1:172.21.15.45:6790/0 conn(0x7f94d4009140 legacy=0x7f94d40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94f0ff9640 1 -- 172.21.15.45:0/4121432017 >> v1:172.21.15.45:6790/0 conn(0x7f94d4009140 legacy=0x7f94d40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:13.356 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:20:13.356 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:13.356 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:13.356 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:20:13.357 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 62, 2024-09-16T03:20:13.357 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 238408, 2024-09-16T03:20:13.357 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:20:13.357 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:20:13.357 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:20:13.357 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:20:13.358 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.358 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:20:13.358 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:13.358 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:20:13.358 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:20:13.358 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:13.358 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:13.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:13.359 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:13.359 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:13.359 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:13.359 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:13.359 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:13.360 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:13.360 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:13.360 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:13.360 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.360 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:20:13.360 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:20:13.361 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:13.361 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:13.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:13.361 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:13.361 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:13.361 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:13.362 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:13.362 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:13.362 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:13.362 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:13.362 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:13.363 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.363 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:20:13.363 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:20:13.363 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:20:13.363 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:20:13.364 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:20:13.364 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:20:13.364 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:13.364 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:13.364 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:20:13.364 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:20:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:20:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:20:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:20:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:20:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:20:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:20:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:20:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:20:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:20:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:20:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:20:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:20:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:20:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:20:13.377 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:20:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 10 librados: watch_flush enter 2024-09-16T03:20:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 10 librados: watch_flush exit 2024-09-16T03:20:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 20 client.144326.objecter shutdown clearing up homeless session... 2024-09-16T03:20:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 10 client.144326.objecter successfully canceled tick 2024-09-16T03:20:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 >> v1:172.21.15.46:6812/149360280 conn(0x7f94e003e310 legacy=0x7f94e00407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 10 monclient: shutdown 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 >> v1:172.21.15.45:6790/0 conn(0x7f94f409fc40 legacy=0x7f94f40727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94f8daf640 1 -- 172.21.15.45:0/4121432017 reap_dead start 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94f8daf640 1 -- 172.21.15.45:0/4121432017 reap_dead start 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 shutdown_connections 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 >> 172.21.15.45:0/4121432017 conn(0x7f94f40a2bd0 msgr2=0x7f94f413ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 shutdown_connections 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 1 -- 172.21.15.45:0/4121432017 wait complete. 2024-09-16T03:20:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.334+0000 7f94fa812640 1 librados: shutdown 2024-09-16T03:20:13.380 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:20:13.381 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:20:13.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:20:13.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient: get_monmap_and_config 2024-09-16T03:20:13.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient: build_initial_monmap 2024-09-16T03:20:13.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient: monmap: 2024-09-16T03:20:13.499 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:13.499 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.499 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:13.501024+0000 2024-09-16T03:20:13.500 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:13.501024+0000 2024-09-16T03:20:13.500 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:13.500 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.500 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:13.500 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:13.500 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:13.500 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 1 Processor -- start 2024-09-16T03:20:13.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 1 -- start start 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient: init 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient(hunting): picked mon.noname-b con 0x7f9b14135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1409fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1413c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1409f7e0 con 0x7f9b1409fc40 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1409b160 con 0x7f9b1413c010 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1409b440 con 0x7f9b14135950 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient(hunting): authenticate will time out at 1904.150302s 2024-09-16T03:20:13.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b137fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9b14135950 0x7f9b14135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43306/0 (socket says 172.21.15.45:43306) 2024-09-16T03:20:13.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b137fe640 1 -- 172.21.15.45:0/436416909 learned_addr learned my addr 172.21.15.45:0/436416909 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:13.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143854059 0 0) 0x7f9b1409b440 con 0x7f9b14135950 2024-09-16T03:20:13.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient(hunting): my global_id is 144335 2024-09-16T03:20:13.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b00003530 con 0x7f9b14135950 2024-09-16T03:20:13.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2065888543 0 0) 0x7f9b1409f7e0 con 0x7f9b1409fc40 2024-09-16T03:20:13.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient(hunting): my global_id is 134433 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b1409b440 con 0x7f9b1409fc40 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3977883444 0 0) 0x7f9b00003530 con 0x7f9b14135950 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 >> v1:172.21.15.46:6789/0 conn(0x7f9b1413c010 legacy=0x7f9b1413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 >> v1:172.21.15.45:6789/0 conn(0x7f9b1409fc40 legacy=0x7f9b1413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: _finish_hunting 0 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 monclient: found mon.noname-b 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b14136440 con 0x7f9b14135950 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: _finish_auth 0 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: _check_auth_tickets 2024-09-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9afc0016a0 con 0x7f9b14135950 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 5 monclient: authenticate success, global_id 144335 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: dump: 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9afc0018c0 con 0x7f9b14135950 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 1 -- 172.21.15.45:0/436416909 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9afc001d10 con 0x7f9b14135950 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b127fc640 10 monclient: dump: 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 4 set_mon_vals no callback set 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 10 monclient: shutdown 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 1 -- 172.21.15.45:0/436416909 >> v1:172.21.15.45:6790/0 conn(0x7f9b14135950 legacy=0x7f9b14135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 1 -- 172.21.15.45:0/436416909 shutdown_connections 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.498+0000 7f9b19eb0640 1 -- 172.21.15.45:0/436416909 >> 172.21.15.45:0/436416909 conn(0x7f9b140a2bd0 msgr2=0x7f9b140a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 -- 172.21.15.45:0/436416909 shutdown_connections 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 -- 172.21.15.45:0/436416909 wait complete. 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient: build_initial_monmap 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient: monmap: 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 librados: starting msgr at 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 librados: starting objecter 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 Processor -- start 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 -- start start 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 librados: setting wanted keys 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 librados: calling monclient init 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient: init 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient(hunting): picked mon.noname-b con 0x7f9b14135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1413c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1413bb20 con 0x7f9b1413c010 2024-09-16T03:20:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1409d2c0 con 0x7f9b14135950 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1413b690 con 0x7f9b1409fc40 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient(hunting): authenticate will time out at 1904.153636s 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b137fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9b1409fc40 0x7f9b141ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43310/0 (socket says 172.21.15.45:43310) 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b137fe640 1 -- 172.21.15.45:0/721248706 learned_addr learned my addr 172.21.15.45:0/721248706 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 client.?.objecter ms_handle_connect 0x7f9b1409fc40 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 client.?.objecter ms_handle_connect 0x7f9b1413c010 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1694021452 0 0) 0x7f9b1413b690 con 0x7f9b1409fc40 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): my global_id is 144338 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9af4004ec0 con 0x7f9b1409fc40 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1614715211 0 0) 0x7f9b1409b160 con 0x7f9b1413c010 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): my global_id is 134436 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b1413b690 con 0x7f9b1413c010 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 client.?.objecter ms_handle_connect 0x7f9b14135950 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 155009169 0 0) 0x7f9af4004ec0 con 0x7f9b1409fc40 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b1409b160 con 0x7f9b1409fc40 2024-09-16T03:20:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9afc001c10 con 0x7f9b1409fc40 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): dump: 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.521 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2790202217 0 0) 0x7f9b1409d2c0 con 0x7f9b14135950 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): my global_id is 164119 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9af4004ec0 con 0x7f9b14135950 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2489577303 0 0) 0x7f9b1413b690 con 0x7f9b1413c010 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b1409d2c0 con 0x7f9b1413c010 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b0c0018c0 con 0x7f9b1413c010 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient(hunting): dump: 2024-09-16T03:20:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1800723347 0 0) 0x7f9b1409b160 con 0x7f9b1409fc40 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 >> v1:172.21.15.46:6789/0 conn(0x7f9b14135950 legacy=0x7f9b140737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 >> v1:172.21.15.45:6789/0 conn(0x7f9b1413c010 legacy=0x7f9b141d4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:20:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 monclient: found mon.c 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1409d7d0 con 0x7f9b1409fc40 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient: _finish_auth 0 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 5 monclient: authenticate success, global_id 144338 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient: _renew_subs 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9b141cd4c0 con 0x7f9b1409fc40 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 client.144338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient: _renew_subs 2024-09-16T03:20:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9b141d64d0 con 0x7f9b1409fc40 2024-09-16T03:20:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 librados: init done 2024-09-16T03:20:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9afc001820 con 0x7f9b1409fc40 2024-09-16T03:20:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9afc004e20 con 0x7f9b1409fc40 2024-09-16T03:20:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b11ffb640 4 set_mon_vals no callback set 2024-09-16T03:20:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 monclient: dump: 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9af80065d0 con 0x7f9af8002870 2024-09-16T03:20:13.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f9afc010e90 con 0x7f9b1409fc40 2024-09-16T03:20:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 client.144338.objecter ms_handle_connect 0x7f9af8002870 2024-09-16T03:20:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 client.144338.objecter resend_mon_ops 2024-09-16T03:20:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(628..628 src has 99..628) ==== 6004+0+0 (unknown 2367076389 0 0) 0x7f9afc04bac0 con 0x7f9b1409fc40 2024-09-16T03:20:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 10 client.144338.objecter ms_dispatch 0x7f9b14000ba0 osd_map(628..628 src has 99..628) 2024-09-16T03:20:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 3 client.144338.objecter handle_osd_map got epochs [628,628] > 0 2024-09-16T03:20:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.502+0000 7f9b0bfff640 3 client.144338.objecter handle_osd_map decoding full epoch 628 2024-09-16T03:20:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 20 client.144338.objecter dump_active .. 0 homeless 2024-09-16T03:20:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b13fff640 10 monclient: get_auth_request con 0x7f9af403e2b0 auth_method 0 2024-09-16T03:20:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161112216 0 0) 0x7f9af80065d0 con 0x7f9af8002870 2024-09-16T03:20:13.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 monclient(hunting): my global_id is 164122 2024-09-16T03:20:13.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b1409b160 con 0x7f9af8002870 2024-09-16T03:20:13.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3346253127 0 0) 0x7f9b1409b160 con 0x7f9af8002870 2024-09-16T03:20:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9af80065d0 con 0x7f9af8002870 2024-09-16T03:20:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b04001b90 con 0x7f9af8002870 2024-09-16T03:20:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:20:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 monclient: dump: 2024-09-16T03:20:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 client.144338.objecter ms_handle_connect 0x7f9af403e2b0 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3543708070 0 0) 0x7f9af80065d0 con 0x7f9af8002870 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9af800a170 con 0x7f9af8002870 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9b04001e50 con 0x7f9af8002870 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 >> v1:172.21.15.46:6789/0 conn(0x7f9af8002870 legacy=0x7f9af8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.506+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 >> v1:172.21.15.46:6789/0 conn(0x7f9af8002870 legacy=0x7f9af8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.510+0000 7f9b19eb0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.510+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9af8009e10 con 0x7f9af80090f0 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 client.144338.objecter ms_handle_connect 0x7f9af80090f0 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 client.144338.objecter resend_mon_ops 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 670384346 0 0) 0x7f9af8009e10 con 0x7f9af80090f0 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 monclient(hunting): my global_id is 164125 2024-09-16T03:20:13.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9af80065d0 con 0x7f9af80090f0 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2888234112 0 0) 0x7f9af80065d0 con 0x7f9af80090f0 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9af8009e10 con 0x7f9af80090f0 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9b04003b40 con 0x7f9af80090f0 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 monclient: dump: 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.533 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.534 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.534 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2122324977 0 0) 0x7f9af8009e10 con 0x7f9af80090f0 2024-09-16T03:20:13.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9af800c3b0 con 0x7f9af80090f0 2024-09-16T03:20:13.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3967 (unknown 0 0 3708075413) 0x7f9b04004ff0 con 0x7f9af80090f0 2024-09-16T03:20:13.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:20:13.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:13.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 >> v1:172.21.15.46:6789/0 conn(0x7f9af80090f0 legacy=0x7f9af8009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b0bfff640 1 -- 172.21.15.45:0/721248706 >> v1:172.21.15.46:6789/0 conn(0x7f9af80090f0 legacy=0x7f9af8009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:13.535 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:20:13.535 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:13.535 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:13.535 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 62, 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 15823, 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:13.538 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:13.539 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:13.540 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:13.541 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:20:13.542 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.544 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:13.544 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:20:13.544 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:13.544 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.544 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:20:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:20:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:20:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:20:13.548 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:20:13.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 10 librados: watch_flush enter 2024-09-16T03:20:13.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 10 librados: watch_flush exit 2024-09-16T03:20:13.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 20 client.144338.objecter shutdown clearing up homeless session... 2024-09-16T03:20:13.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 10 client.144338.objecter successfully canceled tick 2024-09-16T03:20:13.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 >> v1:172.21.15.46:6812/149360280 conn(0x7f9af403e2b0 legacy=0x7f9af4040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 10 monclient: shutdown 2024-09-16T03:20:13.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:13.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 >> v1:172.21.15.45:6790/0 conn(0x7f9b1409fc40 legacy=0x7f9b141ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b13fff640 1 -- 172.21.15.45:0/721248706 reap_dead start 2024-09-16T03:20:13.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b13fff640 1 -- 172.21.15.45:0/721248706 reap_dead start 2024-09-16T03:20:13.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 shutdown_connections 2024-09-16T03:20:13.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 >> 172.21.15.45:0/721248706 conn(0x7f9b140a2bd0 msgr2=0x7f9b1413ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:13.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 shutdown_connections 2024-09-16T03:20:13.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 1 -- 172.21.15.45:0/721248706 wait complete. 2024-09-16T03:20:13.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.514+0000 7f9b19eb0640 1 librados: shutdown 2024-09-16T03:20:13.552 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:20:13.552 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:20:13.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient: get_monmap_and_config 2024-09-16T03:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient: build_initial_monmap 2024-09-16T03:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient: monmap: 2024-09-16T03:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:13.672445+0000 2024-09-16T03:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:13.672445+0000 2024-09-16T03:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 Processor -- start 2024-09-16T03:20:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 -- start start 2024-09-16T03:20:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient: init 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient(hunting): picked mon.noname-b con 0x7f25c80a04f0 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient(hunting): picked mon.noname-c con 0x7f25c8134430 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient(hunting): picked mon.noname-a con 0x7f25c81348c0 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c8074420 con 0x7f25c81348c0 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c809ca90 con 0x7f25c8134430 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c80730d0 con 0x7f25c80a04f0 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient(hunting): authenticate will time out at 1904.322054s 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c6d9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f25c80a04f0 0x7f25c80a08d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43320/0 (socket says 172.21.15.45:43320) 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c6d9d640 1 -- 172.21.15.45:0/3073646372 learned_addr learned my addr 172.21.15.45:0/3073646372 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4117387243 0 0) 0x7f25c80730d0 con 0x7f25c80a04f0 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient(hunting): my global_id is 144341 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25b0003530 con 0x7f25c80a04f0 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572044688 0 0) 0x7f25c8074420 con 0x7f25c81348c0 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient(hunting): my global_id is 134439 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c80730d0 con 0x7f25c81348c0 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1663585106 0 0) 0x7f25b0003530 con 0x7f25c80a04f0 2024-09-16T03:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 >> v1:172.21.15.46:6789/0 conn(0x7f25c8134430 legacy=0x7f25c813ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 >> v1:172.21.15.45:6789/0 conn(0x7f25c81348c0 legacy=0x7f25c81401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 monclient: found mon.noname-b 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c80a4030 con 0x7f25c80a04f0 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: _finish_auth 0 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f25b80016a0 con 0x7f25c80a04f0 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 5 monclient: authenticate success, global_id 144341 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: dump: 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f25b80018c0 con 0x7f25c80a04f0 2024-09-16T03:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 4 set_mon_vals no callback set 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 1 -- 172.21.15.45:0/3073646372 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f25b8001d50 con 0x7f25c80a04f0 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25c5d9b640 10 monclient: dump: 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 10 monclient: shutdown 2024-09-16T03:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 -- 172.21.15.45:0/3073646372 >> v1:172.21.15.45:6790/0 conn(0x7f25c80a04f0 legacy=0x7f25c80a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 -- 172.21.15.45:0/3073646372 shutdown_connections 2024-09-16T03:20:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 -- 172.21.15.45:0/3073646372 >> 172.21.15.45:0/3073646372 conn(0x7f25c80a32b0 msgr2=0x7f25c80a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 -- 172.21.15.45:0/3073646372 shutdown_connections 2024-09-16T03:20:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.670+0000 7f25cd084640 1 -- 172.21.15.45:0/3073646372 wait complete. 2024-09-16T03:20:13.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: build_initial_monmap 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: monmap: 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 librados: starting msgr at 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 librados: starting objecter 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 Processor -- start 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 -- start start 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 librados: setting wanted keys 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 librados: calling monclient init 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: init 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient(hunting): picked mon.noname-b con 0x7f25c80a04f0 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient(hunting): picked mon.noname-a con 0x7f25c8134430 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient(hunting): picked mon.noname-c con 0x7f25c81348c0 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c812fef0 con 0x7f25c8134430 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c80a24a0 con 0x7f25c80a04f0 2024-09-16T03:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c8074c90 con 0x7f25c81348c0 2024-09-16T03:20:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient(hunting): authenticate will time out at 1904.325438s 2024-09-16T03:20:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25c759e640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f25c81348c0 0x7f25c81d4e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43322/0 (socket says 172.21.15.45:43322) 2024-09-16T03:20:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25c759e640 1 -- 172.21.15.45:0/2445919030 learned_addr learned my addr 172.21.15.45:0/2445919030 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25c659c640 1 --1- 172.21.15.45:0/2445919030 >> v1:172.21.15.45:6789/0 conn(0x7f25c8134430 0x7f25c812f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37918/0 (socket says 172.21.15.45:37918) 2024-09-16T03:20:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 client.?.objecter ms_handle_connect 0x7f25c8134430 2024-09-16T03:20:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 client.?.objecter ms_handle_connect 0x7f25c81348c0 2024-09-16T03:20:13.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836398460 0 0) 0x7f25c812fef0 con 0x7f25c8134430 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): my global_id is 134442 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25b4004ec0 con 0x7f25c8134430 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 client.?.objecter ms_handle_connect 0x7f25c80a04f0 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3597712530 0 0) 0x7f25c8074c90 con 0x7f25c81348c0 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): my global_id is 144344 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c812fef0 con 0x7f25c81348c0 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779231415 0 0) 0x7f25c80a24a0 con 0x7f25c80a04f0 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): my global_id is 164131 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c8074c90 con 0x7f25c80a04f0 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3585909094 0 0) 0x7f25b4004ec0 con 0x7f25c8134430 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25c80a24a0 con 0x7f25c8134430 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 607152275 0 0) 0x7f25c812fef0 con 0x7f25c81348c0 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25b4004ec0 con 0x7f25c81348c0 2024-09-16T03:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f25bc001980 con 0x7f25c8134430 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): dump: 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f25c00016a0 con 0x7f25c81348c0 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient(hunting): dump: 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.694 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 371506009 0 0) 0x7f25c80a24a0 con 0x7f25c8134430 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 >> v1:172.21.15.45:6790/0 conn(0x7f25c81348c0 legacy=0x7f25c81d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 >> v1:172.21.15.46:6789/0 conn(0x7f25c80a04f0 legacy=0x7f25c812f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 monclient: found mon.a 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c8136600 con 0x7f25c8134430 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient: _finish_auth 0 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:20:13.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 5 monclient: authenticate success, global_id 134442 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: _renew_subs 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25c81368e0 con 0x7f25c8134430 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 client.134442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: _renew_subs 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f25c8141220 con 0x7f25c8134430 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 librados: init done 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f25bc001cc0 con 0x7f25c8134430 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f25bc003a70 con 0x7f25c8134430 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:13.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25c559a640 4 set_mon_vals no callback set 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 monclient: dump: 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f25bc003c90 con 0x7f25c8134430 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a8005090 con 0x7f25c8134430 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(628..628 src has 99..628) ==== 6004+0+0 (unknown 2367076389 0 0) 0x7f25bc04ba50 con 0x7f25c8134430 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 10 client.134442.objecter ms_dispatch 0x7f25c8000ba0 osd_map(628..628 src has 99..628) 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 3 client.134442.objecter handle_osd_map got epochs [628,628] > 0 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 3 client.134442.objecter handle_osd_map decoding full epoch 628 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.674+0000 7f25af7fe640 20 client.134442.objecter dump_active .. 0 homeless 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.678+0000 7f25c6d9d640 10 monclient: get_auth_request con 0x7f25b403e1e0 auth_method 0 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.678+0000 7f25af7fe640 10 client.134442.objecter ms_handle_connect 0x7f25b403e1e0 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.682+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f25bc016a60 con 0x7f25c8134430 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.682+0000 7f25af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:13.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.682+0000 7f25af7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.918+0000 7f25cd084640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:20:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.918+0000 7f25cd084640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:20:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.918+0000 7f25cd084640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.918+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f25a8005c50 con 0x7f25c8134430 2024-09-16T03:20:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.918+0000 7f25af7fe640 1 -- 172.21.15.45:0/2445919030 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f25bc01ae40 con 0x7f25c8134430 2024-09-16T03:20:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.918+0000 7f25af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:20:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.918+0000 7f25af7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:13.921 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=13,logm=48} crc {auth=3035466629,config=1097430493,health=459779585,kv=3605765656,logm=1065775657}) 2024-09-16T03:20:13.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=837087287,config=1097430493,health=4009975670,kv=4279991801,logm=295674659}) 2024-09-16T03:20:13.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=13,logm=48} crc {auth=3035466629,config=1097430493,health=459779585,kv=3605765656,logm=1065775657}) 2024-09-16T03:20:13.922 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=11,kv=14,logm=50} crc {auth=2363256759,config=4178277379,health=1717358291,kv=992717753,logm=1536497355}) 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 10 librados: watch_flush enter 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 10 librados: watch_flush exit 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 20 client.134442.objecter shutdown clearing up homeless session... 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 10 client.134442.objecter successfully canceled tick 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 >> v1:172.21.15.46:6812/149360280 conn(0x7f25b403e1e0 legacy=0x7f25b40406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 10 monclient: shutdown 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 >> v1:172.21.15.45:6789/0 conn(0x7f25c8134430 legacy=0x7f25c812f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 shutdown_connections 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 >> 172.21.15.45:0/2445919030 conn(0x7f25c80a32b0 msgr2=0x7f25c809f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 shutdown_connections 2024-09-16T03:20:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 1 -- 172.21.15.45:0/2445919030 wait complete. 2024-09-16T03:20:13.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f25cd084640 1 librados: shutdown 2024-09-16T03:20:13.925 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.925 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133546037}) 2024-09-16T03:20:13.925 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3889467375}) 2024-09-16T03:20:13.925 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.925 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133546037}) 2024-09-16T03:20:13.925 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.922+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=953075931}) 2024-09-16T03:20:13.926 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.926+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.926+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3750466761}) 2024-09-16T03:20:13.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.926+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1588310712}) 2024-09-16T03:20:13.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.926+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.926+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3750466761}) 2024-09-16T03:20:13.927 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.926+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481331034}) 2024-09-16T03:20:13.930 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.930+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.930 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.930+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4171222500}) 2024-09-16T03:20:13.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.930+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=802529483}) 2024-09-16T03:20:13.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.930+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.930+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4171222500}) 2024-09-16T03:20:13.931 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.930+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=432929468}) 2024-09-16T03:20:13.937 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.934+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.937 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.934+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455307722}) 2024-09-16T03:20:13.937 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.934+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1440895822}) 2024-09-16T03:20:13.937 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.934+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.937 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.934+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455307722}) 2024-09-16T03:20:13.937 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.934+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1665697155}) 2024-09-16T03:20:13.942 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:20:13.948 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.946+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.948 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.946+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689114757}) 2024-09-16T03:20:13.949 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.946+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1293234910}) 2024-09-16T03:20:13.949 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.946+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.949 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.946+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689114757}) 2024-09-16T03:20:13.949 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.946+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536761916}) 2024-09-16T03:20:13.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.950+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.950+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788687995}) 2024-09-16T03:20:13.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.950+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3651840937}) 2024-09-16T03:20:13.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.950+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.950+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788687995}) 2024-09-16T03:20:13.952 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.950+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=8691620}) 2024-09-16T03:20:13.956 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.954+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.956 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.954+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4054347491}) 2024-09-16T03:20:13.956 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.954+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2720323915}) 2024-09-16T03:20:13.956 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.954+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.956 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.954+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4054347491}) 2024-09-16T03:20:13.957 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.954+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2388487463}) 2024-09-16T03:20:13.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498559976}) 2024-09-16T03:20:13.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1492088453}) 2024-09-16T03:20:13.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498559976}) 2024-09-16T03:20:13.958 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=119694962}) 2024-09-16T03:20:13.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777504545}) 2024-09-16T03:20:13.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551905172}) 2024-09-16T03:20:13.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777504545}) 2024-09-16T03:20:13.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.958+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174792953}) 2024-09-16T03:20:13.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2812791221}) 2024-09-16T03:20:13.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=534925526}) 2024-09-16T03:20:13.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2812791221}) 2024-09-16T03:20:13.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=52684055}) 2024-09-16T03:20:13.964 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.964 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520250637}) 2024-09-16T03:20:13.964 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4023323053}) 2024-09-16T03:20:13.964 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.964 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520250637}) 2024-09-16T03:20:13.964 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2481068325}) 2024-09-16T03:20:13.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13680374}) 2024-09-16T03:20:13.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.962+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=195881751}) 2024-09-16T03:20:13.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13680374}) 2024-09-16T03:20:13.966 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594879020}) 2024-09-16T03:20:13.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026473628}) 2024-09-16T03:20:13.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755905625}) 2024-09-16T03:20:13.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026473628}) 2024-09-16T03:20:13.968 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994160911}) 2024-09-16T03:20:13.969 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.969 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3347570640}) 2024-09-16T03:20:13.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2793320810}) 2024-09-16T03:20:13.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3347570640}) 2024-09-16T03:20:13.970 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.966+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333048097}) 2024-09-16T03:20:13.971 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.971 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354053486}) 2024-09-16T03:20:13.971 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214220238}) 2024-09-16T03:20:13.971 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.971 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354053486}) 2024-09-16T03:20:13.971 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2617430215}) 2024-09-16T03:20:13.973 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.973 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3591823377}) 2024-09-16T03:20:13.973 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272743251}) 2024-09-16T03:20:13.973 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.973 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3591823377}) 2024-09-16T03:20:13.973 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.970+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2611153712}) 2024-09-16T03:20:13.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416452689}) 2024-09-16T03:20:13.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1291451514}) 2024-09-16T03:20:13.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416452689}) 2024-09-16T03:20:13.976 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1486611155}) 2024-09-16T03:20:13.977 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.977 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824147240}) 2024-09-16T03:20:13.977 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=857983551}) 2024-09-16T03:20:13.977 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.977 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824147240}) 2024-09-16T03:20:13.977 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.974+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776421057}) 2024-09-16T03:20:13.979 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.979 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1510933302}) 2024-09-16T03:20:13.979 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1367836314}) 2024-09-16T03:20:13.979 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.979 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1510933302}) 2024-09-16T03:20:13.979 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218471836}) 2024-09-16T03:20:13.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=167481117}) 2024-09-16T03:20:13.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1869566933}) 2024-09-16T03:20:13.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=167481117}) 2024-09-16T03:20:13.981 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.978+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1906856951}) 2024-09-16T03:20:13.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3904246422}) 2024-09-16T03:20:13.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2991215444}) 2024-09-16T03:20:13.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3904246422}) 2024-09-16T03:20:13.983 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188780409}) 2024-09-16T03:20:13.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=995318656}) 2024-09-16T03:20:13.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314561428}) 2024-09-16T03:20:13.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=995318656}) 2024-09-16T03:20:13.985 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.982+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774030628}) 2024-09-16T03:20:13.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236573664}) 2024-09-16T03:20:13.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1093985303}) 2024-09-16T03:20:13.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236573664}) 2024-09-16T03:20:13.987 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2040372772}) 2024-09-16T03:20:13.989 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.989 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168665184}) 2024-09-16T03:20:13.989 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=627945592}) 2024-09-16T03:20:13.989 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.989 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168665184}) 2024-09-16T03:20:13.989 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.986+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696700936}) 2024-09-16T03:20:13.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4214080930}) 2024-09-16T03:20:13.990 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1317691056}) 2024-09-16T03:20:13.991 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.991 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4214080930}) 2024-09-16T03:20:13.991 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650040170}) 2024-09-16T03:20:13.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713196508}) 2024-09-16T03:20:13.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3222238951}) 2024-09-16T03:20:13.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713196508}) 2024-09-16T03:20:13.992 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.990+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=71595417}) 2024-09-16T03:20:13.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090629943}) 2024-09-16T03:20:13.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3891026759}) 2024-09-16T03:20:13.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090629943}) 2024-09-16T03:20:13.994 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1782008130}) 2024-09-16T03:20:13.995 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878204867}) 2024-09-16T03:20:13.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=560229366}) 2024-09-16T03:20:13.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878204867}) 2024-09-16T03:20:13.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=391901732}) 2024-09-16T03:20:13.997 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129111088}) 2024-09-16T03:20:13.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651746146}) 2024-09-16T03:20:13.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129111088}) 2024-09-16T03:20:13.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.994+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2146583638}) 2024-09-16T03:20:13.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:13.999 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373350322}) 2024-09-16T03:20:14.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4219540053}) 2024-09-16T03:20:14.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373350322}) 2024-09-16T03:20:14.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1767407534}) 2024-09-16T03:20:14.001 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.001 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102632350}) 2024-09-16T03:20:14.001 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2529417688}) 2024-09-16T03:20:14.001 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.001 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102632350}) 2024-09-16T03:20:14.001 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:13.998+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3530824857}) 2024-09-16T03:20:14.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470005838}) 2024-09-16T03:20:14.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=967746493}) 2024-09-16T03:20:14.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470005838}) 2024-09-16T03:20:14.005 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1554662695}) 2024-09-16T03:20:14.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485395212}) 2024-09-16T03:20:14.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388097239}) 2024-09-16T03:20:14.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485395212}) 2024-09-16T03:20:14.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.002+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253371993}) 2024-09-16T03:20:14.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059728259}) 2024-09-16T03:20:14.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628966911}) 2024-09-16T03:20:14.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059728259}) 2024-09-16T03:20:14.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=68805427}) 2024-09-16T03:20:14.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366414879}) 2024-09-16T03:20:14.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3676144002}) 2024-09-16T03:20:14.009 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366414879}) 2024-09-16T03:20:14.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1153075461}) 2024-09-16T03:20:14.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347876033}) 2024-09-16T03:20:14.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3735958130}) 2024-09-16T03:20:14.011 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347876033}) 2024-09-16T03:20:14.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.006+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=581929039}) 2024-09-16T03:20:14.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349793859}) 2024-09-16T03:20:14.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2969828895}) 2024-09-16T03:20:14.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349793859}) 2024-09-16T03:20:14.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1041031427}) 2024-09-16T03:20:14.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493808416}) 2024-09-16T03:20:14.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3437995691}) 2024-09-16T03:20:14.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493808416}) 2024-09-16T03:20:14.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.010+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2679928726}) 2024-09-16T03:20:14.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145357110}) 2024-09-16T03:20:14.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3739226952}) 2024-09-16T03:20:14.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145357110}) 2024-09-16T03:20:14.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246103356}) 2024-09-16T03:20:14.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246736492}) 2024-09-16T03:20:14.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1217902462}) 2024-09-16T03:20:14.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246736492}) 2024-09-16T03:20:14.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.014+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=328751702}) 2024-09-16T03:20:14.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649203870}) 2024-09-16T03:20:14.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2868352096}) 2024-09-16T03:20:14.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649203870}) 2024-09-16T03:20:14.020 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1225001423}) 2024-09-16T03:20:14.022 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.022 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252134825}) 2024-09-16T03:20:14.022 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704954600}) 2024-09-16T03:20:14.022 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.022 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252134825}) 2024-09-16T03:20:14.022 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70522994}) 2024-09-16T03:20:14.022 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.018+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396668747}) 2024-09-16T03:20:14.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2218274682}) 2024-09-16T03:20:14.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396668747}) 2024-09-16T03:20:14.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172550467}) 2024-09-16T03:20:14.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774728669}) 2024-09-16T03:20:14.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2152372442}) 2024-09-16T03:20:14.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774728669}) 2024-09-16T03:20:14.025 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.022+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213407099}) 2024-09-16T03:20:14.027 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644094663}) 2024-09-16T03:20:14.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3090580071}) 2024-09-16T03:20:14.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644094663}) 2024-09-16T03:20:14.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1338702948}) 2024-09-16T03:20:14.029 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.029 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130912615}) 2024-09-16T03:20:14.029 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1506493837}) 2024-09-16T03:20:14.029 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.030 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130912615}) 2024-09-16T03:20:14.030 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.026+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3438888760}) 2024-09-16T03:20:14.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3163126642}) 2024-09-16T03:20:14.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1039839817}) 2024-09-16T03:20:14.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3163126642}) 2024-09-16T03:20:14.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2827985543}) 2024-09-16T03:20:14.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660718978}) 2024-09-16T03:20:14.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3290844156}) 2024-09-16T03:20:14.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660718978}) 2024-09-16T03:20:14.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.030+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721705904}) 2024-09-16T03:20:14.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862309282}) 2024-09-16T03:20:14.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3061823874}) 2024-09-16T03:20:14.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862309282}) 2024-09-16T03:20:14.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1109182369}) 2024-09-16T03:20:14.036 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.036 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467392112}) 2024-09-16T03:20:14.036 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=716487326}) 2024-09-16T03:20:14.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467392112}) 2024-09-16T03:20:14.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.034+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973240105}) 2024-09-16T03:20:14.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913657084}) 2024-09-16T03:20:14.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3285266041}) 2024-09-16T03:20:14.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913657084}) 2024-09-16T03:20:14.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483955991}) 2024-09-16T03:20:14.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617797748}) 2024-09-16T03:20:14.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1543211625}) 2024-09-16T03:20:14.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617797748}) 2024-09-16T03:20:14.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039157405}) 2024-09-16T03:20:14.041 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3326864218}) 2024-09-16T03:20:14.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551464762}) 2024-09-16T03:20:14.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.038+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3326864218}) 2024-09-16T03:20:14.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215283473}) 2024-09-16T03:20:14.043 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.043 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615416438}) 2024-09-16T03:20:14.044 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1955291979}) 2024-09-16T03:20:14.044 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.044 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615416438}) 2024-09-16T03:20:14.044 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12339890}) 2024-09-16T03:20:14.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933136073}) 2024-09-16T03:20:14.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2906903753}) 2024-09-16T03:20:14.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933136073}) 2024-09-16T03:20:14.046 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.042+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=667913640}) 2024-09-16T03:20:14.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687937711}) 2024-09-16T03:20:14.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3538315389}) 2024-09-16T03:20:14.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687937711}) 2024-09-16T03:20:14.048 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3208754999}) 2024-09-16T03:20:14.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100451154}) 2024-09-16T03:20:14.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2765441474}) 2024-09-16T03:20:14.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100451154}) 2024-09-16T03:20:14.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.046+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2730964114}) 2024-09-16T03:20:14.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1745088261}) 2024-09-16T03:20:14.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2661544470}) 2024-09-16T03:20:14.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1745088261}) 2024-09-16T03:20:14.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126639189}) 2024-09-16T03:20:14.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346658801}) 2024-09-16T03:20:14.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838738948}) 2024-09-16T03:20:14.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346658801}) 2024-09-16T03:20:14.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.050+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4085992540}) 2024-09-16T03:20:14.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347522665}) 2024-09-16T03:20:14.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2366490004}) 2024-09-16T03:20:14.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347522665}) 2024-09-16T03:20:14.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1414644581}) 2024-09-16T03:20:14.057 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.057 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=647791018}) 2024-09-16T03:20:14.057 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=296322774}) 2024-09-16T03:20:14.057 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.057 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=647791018}) 2024-09-16T03:20:14.057 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.054+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907571091}) 2024-09-16T03:20:14.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792778558}) 2024-09-16T03:20:14.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2430125363}) 2024-09-16T03:20:14.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792778558}) 2024-09-16T03:20:14.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228636391}) 2024-09-16T03:20:14.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2969766262,mdsmap=2883318988}) 2024-09-16T03:20:14.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=4241447195,mdsmap=2287870078,mgr=1140406602}) 2024-09-16T03:20:14.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2969766262,mdsmap=2883318988}) 2024-09-16T03:20:14.062 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.058+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=347164597,mdsmap=2287870078}) 2024-09-16T03:20:14.064 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.062+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.064 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.062+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=9} crc {mdsmap=3140514009,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=2917498545,mon_config_key=1783149646,monmap=127340147,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1918265199,osd_snap=2175596766,osdmap=2978750710}) 2024-09-16T03:20:14.064 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.062+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=20,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=6} crc {mdsmap=3140514009,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=2074831134,mon_config_key=2154011246,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1918265199,osd_snap=2175596766,osdmap=18305710}) 2024-09-16T03:20:14.064 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.062+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.064 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.062+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=9} crc {mdsmap=3140514009,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=2917498545,mon_config_key=1783149646,monmap=127340147,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=1918265199,osd_snap=2175596766,osdmap=2978750710}) 2024-09-16T03:20:14.064 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.062+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=14} crc {mdsmap=3140514009,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=2074831134,mon_config_key=2955048247,monmap=3372108098,nvmeofgw=598337686,osd_metadata=3034489595,osd_pg_creating=1918265199,osd_snap=3803417099,osdmap=3368748961}) 2024-09-16T03:20:14.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1893196992}) 2024-09-16T03:20:14.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1839802514}) 2024-09-16T03:20:14.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1893196992}) 2024-09-16T03:20:14.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2651344474}) 2024-09-16T03:20:14.068 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.068 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3021639105}) 2024-09-16T03:20:14.068 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1991715400}) 2024-09-16T03:20:14.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3021639105}) 2024-09-16T03:20:14.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.066+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1990049184}) 2024-09-16T03:20:14.070 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1015838018}) 2024-09-16T03:20:14.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=842479866}) 2024-09-16T03:20:14.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1015838018}) 2024-09-16T03:20:14.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2709676418}) 2024-09-16T03:20:14.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2725700941}) 2024-09-16T03:20:14.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2996922086}) 2024-09-16T03:20:14.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2725700941}) 2024-09-16T03:20:14.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.070+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3813149009}) 2024-09-16T03:20:14.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.074+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.074+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=517792326}) 2024-09-16T03:20:14.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.074+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2128358740}) 2024-09-16T03:20:14.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.074+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.074+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=517792326}) 2024-09-16T03:20:14.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.074+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1984147695}) 2024-09-16T03:20:14.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.078+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.078+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1843205666}) 2024-09-16T03:20:14.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.078+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4251001669}) 2024-09-16T03:20:14.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.078+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.078+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1843205666}) 2024-09-16T03:20:14.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.078+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4287971936}) 2024-09-16T03:20:14.083 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.082+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.083 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.082+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=216854584}) 2024-09-16T03:20:14.083 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.082+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=941974842}) 2024-09-16T03:20:14.083 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.082+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.083 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.082+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=216854584}) 2024-09-16T03:20:14.083 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.082+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3703914846}) 2024-09-16T03:20:14.095 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.094+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.095 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.094+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=2644918265}) 2024-09-16T03:20:14.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.094+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=533174274}) 2024-09-16T03:20:14.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.094+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:20:14.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.094+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=2644918265}) 2024-09-16T03:20:14.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:20:14.094+0000 7f869e778640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=70} crc {osdmap=96379017}) 2024-09-16T03:20:14.943 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:20:14.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:20:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient: get_monmap_and_config 2024-09-16T03:20:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient: build_initial_monmap 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient: monmap: 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:15.085587+0000 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:15.085587+0000 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 1 Processor -- start 2024-09-16T03:20:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 1 -- start start 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient: init 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient(hunting): picked mon.noname-c con 0x7f9a6c137c90 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient(hunting): picked mon.noname-a con 0x7f9a6c074bc0 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient(hunting): picked mon.noname-b con 0x7f9a6c13e310 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c09da80 con 0x7f9a6c074bc0 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c0a5c40 con 0x7f9a6c137c90 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c09dcc0 con 0x7f9a6c13e310 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 10 monclient(hunting): authenticate will time out at 1905.735020s 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6ad9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9a6c13e310 0x7f9a6c140700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43328/0 (socket says 172.21.15.45:43328) 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6ad9d640 1 -- 172.21.15.45:0/1955895737 learned_addr learned my addr 172.21.15.45:0/1955895737 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3248460178 0 0) 0x7f9a6c09da80 con 0x7f9a6c074bc0 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient(hunting): my global_id is 134445 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a58003530 con 0x7f9a6c074bc0 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1751596600 0 0) 0x7f9a6c09dcc0 con 0x7f9a6c13e310 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient(hunting): my global_id is 144347 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a6c09da80 con 0x7f9a6c13e310 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3067594661 0 0) 0x7f9a58003530 con 0x7f9a6c074bc0 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 >> v1:172.21.15.45:6790/0 conn(0x7f9a6c13e310 legacy=0x7f9a6c140700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 >> v1:172.21.15.46:6789/0 conn(0x7f9a6c137c90 legacy=0x7f9a6c138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient: _finish_hunting 0 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 monclient: found mon.noname-a 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c073260 con 0x7f9a6c074bc0 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient: _finish_auth 0 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient: _check_auth_tickets 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9a540016a0 con 0x7f9a6c074bc0 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 5 monclient: authenticate success, global_id 134445 2024-09-16T03:20:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a70bf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:15.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.082+0000 7f9a6959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a6959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:20:15.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a6959a640 10 monclient: dump: 2024-09-16T03:20:15.092 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.092 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.092 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.092 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.092 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.092 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9a540018c0 con 0x7f9a6c074bc0 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a6959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a6959a640 1 -- 172.21.15.45:0/1955895737 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9a54001cd0 con 0x7f9a6c074bc0 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a6959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a6959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a6959a640 10 monclient: dump: 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 4 set_mon_vals no callback set 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: shutdown 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- 172.21.15.45:0/1955895737 >> v1:172.21.15.45:6789/0 conn(0x7f9a6c074bc0 legacy=0x7f9a6c13cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- 172.21.15.45:0/1955895737 shutdown_connections 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- 172.21.15.45:0/1955895737 >> 172.21.15.45:0/1955895737 conn(0x7f9a6c0aa510 msgr2=0x7f9a6c0aa920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- 172.21.15.45:0/1955895737 shutdown_connections 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- 172.21.15.45:0/1955895737 wait complete. 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: build_initial_monmap 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: monmap: 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.095 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 librados: starting msgr at 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 librados: starting objecter 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 Processor -- start 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- start start 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 librados: setting wanted keys 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 librados: calling monclient init 2024-09-16T03:20:15.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: init 2024-09-16T03:20:15.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient(hunting): picked mon.noname-c con 0x7f9a6c074bc0 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient(hunting): picked mon.noname-a con 0x7f9a6c137c90 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient(hunting): picked mon.noname-b con 0x7f9a6c13e310 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c13c350 con 0x7f9a6c137c90 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c13bf30 con 0x7f9a6c13e310 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c0a1300 con 0x7f9a6c074bc0 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient(hunting): authenticate will time out at 1905.738354s 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a69d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9a6c137c90 0x7f9a6c1390d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37928/0 (socket says 172.21.15.45:37928) 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a69d9b640 1 -- 172.21.15.45:0/2152090092 learned_addr learned my addr 172.21.15.45:0/2152090092 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a6a59c640 1 --1- 172.21.15.45:0/2152090092 >> v1:172.21.15.45:6790/0 conn(0x7f9a6c074bc0 0x7f9a6c1389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43330/0 (socket says 172.21.15.45:43330) 2024-09-16T03:20:15.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 client.?.objecter ms_handle_connect 0x7f9a6c074bc0 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 client.?.objecter ms_handle_connect 0x7f9a6c137c90 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422660367 0 0) 0x7f9a6c0a1300 con 0x7f9a6c074bc0 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): my global_id is 144350 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a4c004ef0 con 0x7f9a6c074bc0 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3160050144 0 0) 0x7f9a6c073260 con 0x7f9a6c137c90 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): my global_id is 134448 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a6c0a1300 con 0x7f9a6c137c90 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 client.?.objecter ms_handle_connect 0x7f9a6c13e310 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1326377580 0 0) 0x7f9a4c004ef0 con 0x7f9a6c074bc0 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a6c073260 con 0x7f9a6c074bc0 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9a5c001980 con 0x7f9a6c074bc0 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): dump: 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 610002552 0 0) 0x7f9a6c0a1300 con 0x7f9a6c137c90 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a4c004ef0 con 0x7f9a6c137c90 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9a54001c10 con 0x7f9a6c137c90 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): dump: 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3747174331 0 0) 0x7f9a6c13bf30 con 0x7f9a6c13e310 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient(hunting): my global_id is 164137 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a6c0a1300 con 0x7f9a6c13e310 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 441652830 0 0) 0x7f9a6c073260 con 0x7f9a6c074bc0 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 >> v1:172.21.15.46:6789/0 conn(0x7f9a6c13e310 legacy=0x7f9a6c1d4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 >> v1:172.21.15.45:6789/0 conn(0x7f9a6c137c90 legacy=0x7f9a6c1390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 monclient: found mon.c 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c1397e0 con 0x7f9a6c074bc0 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient: _finish_auth 0 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:20:15.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 5 monclient: authenticate success, global_id 144350 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: _renew_subs 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6c139bd0 con 0x7f9a6c074bc0 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 client.144350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: _renew_subs 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9a6c13d720 con 0x7f9a6c074bc0 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a70bf1640 1 librados: init done 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9a5c003620 con 0x7f9a6c074bc0 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9a5c003970 con 0x7f9a6c074bc0 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a62ffd640 10 monclient: dump: 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.086+0000 7f9a68d99640 4 set_mon_vals no callback set 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a70bf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:15.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a70bf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a70bf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f9a5c003b10 con 0x7f9a6c074bc0 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a50005090 con 0x7f9a6c074bc0 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(630..630 src has 99..630) ==== 5917+0+0 (unknown 920938494 0 0) 0x7f9a5c04b8a0 con 0x7f9a6c074bc0 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a62ffd640 10 client.144350.objecter ms_dispatch 0x7f9a6c000ba0 osd_map(630..630 src has 99..630) 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a62ffd640 3 client.144350.objecter handle_osd_map got epochs [630,630] > 0 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a62ffd640 3 client.144350.objecter handle_osd_map decoding full epoch 630 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a62ffd640 20 client.144350.objecter dump_active .. 0 homeless 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a69d9b640 10 monclient: get_auth_request con 0x7f9a4c03e2c0 auth_method 0 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.090+0000 7f9a62ffd640 10 client.144350.objecter ms_handle_connect 0x7f9a4c03e2c0 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.094+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f9a5c016990 con 0x7f9a6c074bc0 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.094+0000 7f9a62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:15.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.094+0000 7f9a62ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:15.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:20:15.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:15.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:15.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9a50005380 con 0x7f9a6c074bc0 2024-09-16T03:20:15.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a62ffd640 1 -- 172.21.15.45:0/2152090092 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 1924395747) 0x7f9a5c016290 con 0x7f9a6c074bc0 2024-09-16T03:20:15.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:15.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a62ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:15.332 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:20:15.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 10 librados: watch_flush enter 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 10 librados: watch_flush exit 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 20 client.144350.objecter shutdown clearing up homeless session... 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 10 client.144350.objecter successfully canceled tick 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 >> v1:172.21.15.46:6812/149360280 conn(0x7f9a4c03e2c0 legacy=0x7f9a4c040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 10 monclient: shutdown 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 >> v1:172.21.15.45:6790/0 conn(0x7f9a6c074bc0 legacy=0x7f9a6c1389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 shutdown_connections 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.330+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 >> 172.21.15.45:0/2152090092 conn(0x7f9a6c0aa510 msgr2=0x7f9a6c0a3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.334+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 shutdown_connections 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.334+0000 7f9a70bf1640 1 -- 172.21.15.45:0/2152090092 wait complete. 2024-09-16T03:20:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.334+0000 7f9a70bf1640 1 librados: shutdown 2024-09-16T03:20:15.350 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:20:15.350 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:20:15.350 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:20:15.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:20:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient: get_monmap_and_config 2024-09-16T03:20:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient: build_initial_monmap 2024-09-16T03:20:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient: monmap: 2024-09-16T03:20:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:15.480443+0000 2024-09-16T03:20:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:15.480443+0000 2024-09-16T03:20:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:15.479 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 Processor -- start 2024-09-16T03:20:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 -- start start 2024-09-16T03:20:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient: init 2024-09-16T03:20:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:20:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient(hunting): picked mon.noname-b con 0x7f4a0c135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient(hunting): picked mon.noname-c con 0x7f4a0c09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient(hunting): picked mon.noname-a con 0x7f4a0c13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0c09f7e0 con 0x7f4a0c13c010 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0c09b160 con 0x7f4a0c09fc40 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0c09b440 con 0x7f4a0c135950 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient(hunting): authenticate will time out at 1906.129744s 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a0b59e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4a0c13c010 0x7f4a0c13e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37930/0 (socket says 172.21.15.45:37930) 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a0b59e640 1 -- 172.21.15.45:0/1502512609 learned_addr learned my addr 172.21.15.45:0/1502512609 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481239791 0 0) 0x7f4a0c09b440 con 0x7f4a0c135950 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient(hunting): my global_id is 144353 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49f4003530 con 0x7f4a0c135950 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3139690772 0 0) 0x7f4a0c09f7e0 con 0x7f4a0c13c010 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient(hunting): my global_id is 134451 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a0c09b440 con 0x7f4a0c13c010 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1064568783 0 0) 0x7f49f4003530 con 0x7f4a0c135950 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 >> v1:172.21.15.46:6789/0 conn(0x7f4a0c09fc40 legacy=0x7f4a0c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 >> v1:172.21.15.45:6789/0 conn(0x7f4a0c13c010 legacy=0x7f4a0c13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:20:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 monclient: found mon.noname-b 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c136440 con 0x7f4a0c135950 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: _finish_auth 0 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f49fc0016a0 con 0x7f4a0c135950 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: dump: 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 5 monclient: authenticate success, global_id 144353 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f49fc001840 con 0x7f4a0c135950 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 1 -- 172.21.15.45:0/1502512609 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f49fc001cd0 con 0x7f4a0c135950 2024-09-16T03:20:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 4 set_mon_vals no callback set 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a09d9b640 10 monclient: dump: 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.487 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 10 monclient: shutdown 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 -- 172.21.15.45:0/1502512609 >> v1:172.21.15.45:6790/0 conn(0x7f4a0c135950 legacy=0x7f4a0c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 -- 172.21.15.45:0/1502512609 shutdown_connections 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 -- 172.21.15.45:0/1502512609 >> 172.21.15.45:0/1502512609 conn(0x7f4a0c0a2bd0 msgr2=0x7f4a0c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 -- 172.21.15.45:0/1502512609 shutdown_connections 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.478+0000 7f4a11178640 1 -- 172.21.15.45:0/1502512609 wait complete. 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient: build_initial_monmap 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient: monmap: 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 librados: starting msgr at 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 librados: starting objecter 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 Processor -- start 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 -- start start 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 librados: setting wanted keys 2024-09-16T03:20:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 librados: calling monclient init 2024-09-16T03:20:15.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient: init 2024-09-16T03:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient(hunting): picked mon.noname-b con 0x7f4a0c09fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient(hunting): picked mon.noname-c con 0x7f4a0c135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient(hunting): picked mon.noname-a con 0x7f4a0c13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0c073890 con 0x7f4a0c13c010 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0c1cd970 con 0x7f4a0c09fc40 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0c13f500 con 0x7f4a0c135950 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient(hunting): authenticate will time out at 1906.133142s 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a0a59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4a0c135950 0x7f4a0c1cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43346/0 (socket says 172.21.15.45:43346) 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a0b59e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4a0c13c010 0x7f4a0c1d4db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37938/0 (socket says 172.21.15.45:37938) 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a0a59c640 1 -- 172.21.15.45:0/3252008239 learned_addr learned my addr 172.21.15.45:0/3252008239 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 client.?.objecter ms_handle_connect 0x7f4a0c135950 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 client.?.objecter ms_handle_connect 0x7f4a0c13c010 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1872919941 0 0) 0x7f4a0c13f500 con 0x7f4a0c135950 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): my global_id is 144356 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49f8004ec0 con 0x7f4a0c135950 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1231807459 0 0) 0x7f4a0c073890 con 0x7f4a0c13c010 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): my global_id is 134454 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a0c13f500 con 0x7f4a0c13c010 2024-09-16T03:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 client.?.objecter ms_handle_connect 0x7f4a0c09fc40 2024-09-16T03:20:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1605232387 0 0) 0x7f49f8004ec0 con 0x7f4a0c135950 2024-09-16T03:20:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a0c073890 con 0x7f4a0c135950 2024-09-16T03:20:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4a00001900 con 0x7f4a0c135950 2024-09-16T03:20:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): dump: 2024-09-16T03:20:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1569100259 0 0) 0x7f4a0c13f500 con 0x7f4a0c13c010 2024-09-16T03:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49f8004ec0 con 0x7f4a0c13c010 2024-09-16T03:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4a04001740 con 0x7f4a0c13c010 2024-09-16T03:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): dump: 2024-09-16T03:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.502 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2848354076 0 0) 0x7f4a0c1cd970 con 0x7f4a0c09fc40 2024-09-16T03:20:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): my global_id is 164143 2024-09-16T03:20:15.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a0c13f500 con 0x7f4a0c09fc40 2024-09-16T03:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 445386419 0 0) 0x7f4a0c073890 con 0x7f4a0c135950 2024-09-16T03:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 >> v1:172.21.15.46:6789/0 conn(0x7f4a0c09fc40 legacy=0x7f4a0c1ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 >> v1:172.21.15.45:6789/0 conn(0x7f4a0c13c010 legacy=0x7f4a0c1d4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 monclient: found mon.c 2024-09-16T03:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c0a2770 con 0x7f4a0c135950 2024-09-16T03:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: _finish_auth 0 2024-09-16T03:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 5 monclient: authenticate success, global_id 144356 2024-09-16T03:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient: _renew_subs 2024-09-16T03:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0c09d2c0 con 0x7f4a0c135950 2024-09-16T03:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 client.144356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient: _renew_subs 2024-09-16T03:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4a0c0735b0 con 0x7f4a0c135950 2024-09-16T03:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 librados: init done 2024-09-16T03:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a00003620 con 0x7f4a0c135950 2024-09-16T03:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4a00003a70 con 0x7f4a0c135950 2024-09-16T03:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a0959a640 4 set_mon_vals no callback set 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: dump: 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f49ec0065d0 con 0x7f49ec002870 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 client.144356.objecter ms_handle_connect 0x7f49ec002870 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 client.144356.objecter resend_mon_ops 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4a00003c10 con 0x7f4a0c135950 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(630..630 src has 99..630) ==== 5917+0+0 (unknown 920938494 0 0) 0x7f4a0004b900 con 0x7f4a0c135950 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 client.144356.objecter ms_dispatch 0x7f4a0c000ba0 osd_map(630..630 src has 99..630) 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 3 client.144356.objecter handle_osd_map got epochs [630,630] > 0 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 3 client.144356.objecter handle_osd_map decoding full epoch 630 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 20 client.144356.objecter dump_active .. 0 homeless 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3185542592 0 0) 0x7f49ec0065d0 con 0x7f49ec002870 2024-09-16T03:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient(hunting): my global_id is 134457 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a0c073890 con 0x7f49ec002870 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 662799607 0 0) 0x7f4a0c073890 con 0x7f49ec002870 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49ec0065d0 con 0x7f49ec002870 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f49fc001910 con 0x7f49ec002870 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f49f37fe640 10 monclient: dump: 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.482+0000 7f4a0b59e640 10 monclient: get_auth_request con 0x7f49f803e260 auth_method 0 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.486+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 663545163 0 0) 0x7f49ec0065d0 con 0x7f49ec002870 2024-09-16T03:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.486+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49ec00a170 con 0x7f49ec002870 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.486+0000 7f49f37fe640 10 client.144356.objecter ms_handle_connect 0x7f49f803e260 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.486+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f49fc001b30 con 0x7f49ec002870 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.486+0000 7f49f37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.486+0000 7f49f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.486+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 >> v1:172.21.15.45:6789/0 conn(0x7f49ec002870 legacy=0x7f49ec004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.486+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 >> v1:172.21.15.45:6789/0 conn(0x7f49ec002870 legacy=0x7f49ec004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f4a11178640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f49ec009e80 con 0x7f49ec009140 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 client.144356.objecter ms_handle_connect 0x7f49ec009140 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 client.144356.objecter resend_mon_ops 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2712426925 0 0) 0x7f49ec009e80 con 0x7f49ec009140 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 monclient(hunting): my global_id is 134460 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49ec0065d0 con 0x7f49ec009140 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4205925176 0 0) 0x7f49ec0065d0 con 0x7f49ec009140 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49ec009e80 con 0x7f49ec009140 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f49fc003860 con 0x7f49ec009140 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 monclient: dump: 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.511 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1794169237 0 0) 0x7f49ec009e80 con 0x7f49ec009140 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f49ec00c420 con 0x7f49ec009140 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4304 (unknown 0 0 1220237407) 0x7f49fc0037f0 con 0x7f49ec009140 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 >> v1:172.21.15.45:6789/0 conn(0x7f49ec009140 legacy=0x7f49ec0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.490+0000 7f49f37fe640 1 -- 172.21.15.45:0/3252008239 >> v1:172.21.15.45:6789/0 conn(0x7f49ec009140 legacy=0x7f49ec0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:15.513 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:20:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:20:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 62, 2024-09-16T03:20:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 348719, 2024-09-16T03:20:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:20:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:20:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:20:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:20:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:20:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:20:15.525 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 10 librados: watch_flush enter 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 10 librados: watch_flush exit 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 20 client.144356.objecter shutdown clearing up homeless session... 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 10 client.144356.objecter successfully canceled tick 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 >> v1:172.21.15.46:6812/149360280 conn(0x7f49f803e260 legacy=0x7f49f8040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 10 monclient: shutdown 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 >> v1:172.21.15.45:6790/0 conn(0x7f4a0c135950 legacy=0x7f4a0c1cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a0b59e640 1 -- 172.21.15.45:0/3252008239 reap_dead start 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a0b59e640 1 -- 172.21.15.45:0/3252008239 reap_dead start 2024-09-16T03:20:15.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 shutdown_connections 2024-09-16T03:20:15.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 >> 172.21.15.45:0/3252008239 conn(0x7f4a0c0a2bd0 msgr2=0x7f4a0c0a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:15.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 shutdown_connections 2024-09-16T03:20:15.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 1 -- 172.21.15.45:0/3252008239 wait complete. 2024-09-16T03:20:15.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.494+0000 7f4a11178640 1 librados: shutdown 2024-09-16T03:20:15.528 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:20:15.528 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:20:15.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:20:15.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient: get_monmap_and_config 2024-09-16T03:20:15.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient: build_initial_monmap 2024-09-16T03:20:15.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient: monmap: 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:15.645698+0000 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:15.645698+0000 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:15.644 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 1 Processor -- start 2024-09-16T03:20:15.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 1 -- start start 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient: init 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient(hunting): picked mon.noname-b con 0x7f61500a25d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient(hunting): picked mon.noname-a con 0x7f615009a850 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient(hunting): picked mon.noname-c con 0x7f615009ac60 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615009c3f0 con 0x7f615009a850 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615009c6d0 con 0x7f615009ac60 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615009c9b0 con 0x7f61500a25d0 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:15.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f61552e6640 10 monclient(hunting): authenticate will time out at 1906.295790s 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f614e7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f615009a850 0x7f6150139250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37966/0 (socket says 172.21.15.45:37966) 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.642+0000 7f614effd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f61500a25d0 0x7f615009a140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43362/0 (socket says 172.21.15.45:43362) 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614e7fc640 1 -- 172.21.15.45:0/1009198781 learned_addr learned my addr 172.21.15.45:0/1009198781 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2789922991 0 0) 0x7f615009c9b0 con 0x7f61500a25d0 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient(hunting): my global_id is 144359 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6138003530 con 0x7f61500a25d0 2024-09-16T03:20:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799525154 0 0) 0x7f615009c3f0 con 0x7f615009a850 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient(hunting): my global_id is 134463 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f615009c9b0 con 0x7f615009a850 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2457467762 0 0) 0x7f615009c9b0 con 0x7f615009a850 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 >> v1:172.21.15.45:6790/0 conn(0x7f61500a25d0 legacy=0x7f615009a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 >> v1:172.21.15.46:6789/0 conn(0x7f615009ac60 legacy=0x7f615013c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 monclient: found mon.noname-a 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615009b350 con 0x7f615009a850 2024-09-16T03:20:15.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: _finish_auth 0 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6144001680 con 0x7f615009a850 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 5 monclient: authenticate success, global_id 134463 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: dump: 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.651 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.652 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.652 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.652 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.652 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:15.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6144001820 con 0x7f615009a850 2024-09-16T03:20:15.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:15.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 1 -- 172.21.15.45:0/1009198781 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6144001c30 con 0x7f615009a850 2024-09-16T03:20:15.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 4 set_mon_vals no callback set 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614dffb640 10 monclient: dump: 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.653 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient: shutdown 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 -- 172.21.15.45:0/1009198781 >> v1:172.21.15.45:6789/0 conn(0x7f615009a850 legacy=0x7f6150139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 -- 172.21.15.45:0/1009198781 shutdown_connections 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 -- 172.21.15.45:0/1009198781 >> 172.21.15.45:0/1009198781 conn(0x7f61500a2bd0 msgr2=0x7f61500a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 -- 172.21.15.45:0/1009198781 shutdown_connections 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 -- 172.21.15.45:0/1009198781 wait complete. 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient: build_initial_monmap 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient: monmap: 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:15.655 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:15.655 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:15.655 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.655 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:15.655 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:15.655 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:15.655 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 librados: starting msgr at 2024-09-16T03:20:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 librados: starting objecter 2024-09-16T03:20:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 Processor -- start 2024-09-16T03:20:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 -- start start 2024-09-16T03:20:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 librados: setting wanted keys 2024-09-16T03:20:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 librados: calling monclient init 2024-09-16T03:20:15.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient: init 2024-09-16T03:20:15.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:15.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:20:15.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient(hunting): picked mon.noname-a con 0x7f615009a850 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:15.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient(hunting): picked mon.noname-c con 0x7f615009ac60 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient(hunting): picked mon.noname-b con 0x7f61500a25d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6150097880 con 0x7f615009a850 2024-09-16T03:20:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6150095010 con 0x7f61500a25d0 2024-09-16T03:20:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615009a640 con 0x7f615009ac60 2024-09-16T03:20:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61552e6640 10 monclient(hunting): authenticate will time out at 1906.299101s 2024-09-16T03:20:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f615009a850 0x7f61500941f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37972/0 (socket says 172.21.15.45:37972) 2024-09-16T03:20:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614effd640 1 -- 172.21.15.45:0/2836255647 learned_addr learned my addr 172.21.15.45:0/2836255647 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f614e7fc640 1 --1- 172.21.15.45:0/2836255647 >> v1:172.21.15.45:6790/0 conn(0x7f615009ac60 0x7f6150094900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43378/0 (socket says 172.21.15.45:43378) 2024-09-16T03:20:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 client.?.objecter ms_handle_connect 0x7f615009a850 2024-09-16T03:20:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 client.?.objecter ms_handle_connect 0x7f615009ac60 2024-09-16T03:20:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3484235780 0 0) 0x7f6150097880 con 0x7f615009a850 2024-09-16T03:20:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): my global_id is 134466 2024-09-16T03:20:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f613c004ec0 con 0x7f615009a850 2024-09-16T03:20:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1159703636 0 0) 0x7f615009a640 con 0x7f615009ac60 2024-09-16T03:20:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): my global_id is 144362 2024-09-16T03:20:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6150097880 con 0x7f615009ac60 2024-09-16T03:20:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 client.?.objecter ms_handle_connect 0x7f61500a25d0 2024-09-16T03:20:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2694969075 0 0) 0x7f613c004ec0 con 0x7f615009a850 2024-09-16T03:20:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f615009a640 con 0x7f615009a850 2024-09-16T03:20:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f61400013a0 con 0x7f615009a850 2024-09-16T03:20:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): dump: 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.671 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1489510768 0 0) 0x7f6150097880 con 0x7f615009ac60 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f613c004ec0 con 0x7f615009ac60 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6144001b70 con 0x7f615009ac60 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): dump: 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1939318214 0 0) 0x7f6150095010 con 0x7f61500a25d0 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 10 monclient(hunting): my global_id is 164149 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.646+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6150097880 con 0x7f61500a25d0 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3586218994 0 0) 0x7f615009a640 con 0x7f615009a850 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 >> v1:172.21.15.45:6790/0 conn(0x7f615009ac60 legacy=0x7f6150094900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 >> v1:172.21.15.46:6789/0 conn(0x7f61500a25d0 legacy=0x7f61501d4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: _finish_hunting 0 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 monclient: found mon.a 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615009a0f0 con 0x7f615009a850 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: _finish_auth 0 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: _check_auth_tickets 2024-09-16T03:20:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 5 monclient: authenticate success, global_id 134466 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 10 monclient: _renew_subs 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6150097e40 con 0x7f615009a850 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 10 client.134466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 10 monclient: _renew_subs 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6150096380 con 0x7f615009a850 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 1 librados: init done 2024-09-16T03:20:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61400047f0 con 0x7f615009a850 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6140004b80 con 0x7f615009a850 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: dump: 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f614d7fa640 4 set_mon_vals no callback set 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61300065d0 con 0x7f6130002870 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 client.134466.objecter ms_handle_connect 0x7f6130002870 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 client.134466.objecter resend_mon_ops 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f6140010bf0 con 0x7f615009a850 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3884017890 0 0) 0x7f61300065d0 con 0x7f6130002870 2024-09-16T03:20:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient(hunting): my global_id is 144365 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f615009a640 con 0x7f6130002870 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(630..630 src has 99..630) ==== 5917+0+0 (unknown 920938494 0 0) 0x7f614004ba40 con 0x7f615009a850 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 client.134466.objecter ms_dispatch 0x7f6150000ba0 osd_map(630..630 src has 99..630) 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 3 client.134466.objecter handle_osd_map got epochs [630,630] > 0 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 3 client.134466.objecter handle_osd_map decoding full epoch 630 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 20 client.134466.objecter dump_active .. 0 homeless 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3860503643 0 0) 0x7f615009a640 con 0x7f6130002870 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61300065d0 con 0x7f6130002870 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6144004c90 con 0x7f6130002870 2024-09-16T03:20:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: dump: 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f614f7fe640 10 monclient: get_auth_request con 0x7f613c03e1a0 auth_method 0 2024-09-16T03:20:15.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2175972778 0 0) 0x7f61300065d0 con 0x7f6130002870 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f613000a170 con 0x7f6130002870 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 client.134466.objecter ms_handle_connect 0x7f613c03e1a0 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f61440060c0 con 0x7f6130002870 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 >> v1:172.21.15.45:6790/0 conn(0x7f6130002870 legacy=0x7f6130004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.650+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 >> v1:172.21.15.45:6790/0 conn(0x7f6130002870 legacy=0x7f6130004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6130009e80 con 0x7f6130009140 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 client.134466.objecter ms_handle_connect 0x7f6130009140 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 client.134466.objecter resend_mon_ops 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3956818811 0 0) 0x7f6130009e80 con 0x7f6130009140 2024-09-16T03:20:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 monclient(hunting): my global_id is 144368 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61300065d0 con 0x7f6130009140 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 262469910 0 0) 0x7f61300065d0 con 0x7f6130009140 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6130009e80 con 0x7f6130009140 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f61440067c0 con 0x7f6130009140 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 monclient: dump: 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3503343103 0 0) 0x7f6130009e80 con 0x7f6130009140 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f613000c420 con 0x7f6130009140 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 1647283712) 0x7f6144006690 con 0x7f6130009140 2024-09-16T03:20:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:20:15.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:15.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 >> v1:172.21.15.45:6790/0 conn(0x7f6130009140 legacy=0x7f61300095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61377fe640 1 -- 172.21.15.45:0/2836255647 >> v1:172.21.15.45:6790/0 conn(0x7f6130009140 legacy=0x7f61300095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:15.683 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:20:15.683 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:15.683 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:15.683 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:20:15.683 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 62, 2024-09-16T03:20:15.683 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 240732, 2024-09-16T03:20:15.684 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:20:15.684 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:20:15.684 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:20:15.684 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:20:15.684 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.684 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 3, 2024-09-16T03:20:15.684 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:15.684 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:20:15.684 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.685 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:15.686 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:20:15.687 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:15.688 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:15.689 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:15.690 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.692 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:20:15.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.696 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:20:15.696 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:20:15.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 10 librados: watch_flush enter 2024-09-16T03:20:15.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 10 librados: watch_flush exit 2024-09-16T03:20:15.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 20 client.134466.objecter shutdown clearing up homeless session... 2024-09-16T03:20:15.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 10 client.134466.objecter successfully canceled tick 2024-09-16T03:20:15.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 >> v1:172.21.15.46:6812/149360280 conn(0x7f613c03e1a0 legacy=0x7f613c040660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 10 monclient: shutdown 2024-09-16T03:20:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 >> v1:172.21.15.45:6789/0 conn(0x7f615009a850 legacy=0x7f61500941f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f614f7fe640 1 -- 172.21.15.45:0/2836255647 reap_dead start 2024-09-16T03:20:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f614f7fe640 1 -- 172.21.15.45:0/2836255647 reap_dead start 2024-09-16T03:20:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 shutdown_connections 2024-09-16T03:20:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 >> 172.21.15.45:0/2836255647 conn(0x7f61500a2bd0 msgr2=0x7f61500a4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 shutdown_connections 2024-09-16T03:20:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 1 -- 172.21.15.45:0/2836255647 wait complete. 2024-09-16T03:20:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.658+0000 7f61552e6640 1 librados: shutdown 2024-09-16T03:20:15.700 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:20:15.700 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:20:15.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:20:15.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient: get_monmap_and_config 2024-09-16T03:20:15.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient: build_initial_monmap 2024-09-16T03:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient: monmap: 2024-09-16T03:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:15.820662+0000 2024-09-16T03:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:15.820662+0000 2024-09-16T03:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:15.820 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.820 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:15.820 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:15.820 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:15.820 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 Processor -- start 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 -- start start 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient: init 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient(hunting): picked mon.noname-c con 0x7f02fc137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient(hunting): picked mon.noname-a con 0x7f02fc138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient(hunting): picked mon.noname-b con 0x7f02fc13e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02fc0a4f90 con 0x7f02fc138100 2024-09-16T03:20:15.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02fc0a4cb0 con 0x7f02fc137d20 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02fc09dc70 con 0x7f02fc13e2a0 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient(hunting): authenticate will time out at 1906.470183s 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02fb7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f02fc138100 0x7f02fc13cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37974/0 (socket says 172.21.15.45:37974) 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02fb7fe640 1 -- 172.21.15.45:0/1309324875 learned_addr learned my addr 172.21.15.45:0/1309324875 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f0300b74640 1 --1- 172.21.15.45:0/1309324875 >> v1:172.21.15.45:6790/0 conn(0x7f02fc13e2a0 0x7f02fc140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43402/0 (socket says 172.21.15.45:43402) 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2295101064 0 0) 0x7f02fc0a4f90 con 0x7f02fc138100 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): my global_id is 134469 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02e4003530 con 0x7f02fc138100 2024-09-16T03:20:15.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3030353461 0 0) 0x7f02fc09dc70 con 0x7f02fc13e2a0 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): my global_id is 144371 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02fc0a4f90 con 0x7f02fc13e2a0 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2729118421 0 0) 0x7f02fc0a4cb0 con 0x7f02fc137d20 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient(hunting): my global_id is 164152 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02fc09dc70 con 0x7f02fc137d20 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 359116283 0 0) 0x7f02e4003530 con 0x7f02fc138100 2024-09-16T03:20:15.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 >> v1:172.21.15.45:6790/0 conn(0x7f02fc13e2a0 legacy=0x7f02fc140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 >> v1:172.21.15.46:6789/0 conn(0x7f02fc137d20 legacy=0x7f02fc0a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: _finish_hunting 0 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 monclient: found mon.noname-a 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc09cc50 con 0x7f02fc138100 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: _finish_auth 0 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: _check_auth_tickets 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f02f0001a20 con 0x7f02fc138100 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: dump: 2024-09-16T03:20:15.834 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 5 monclient: authenticate success, global_id 134469 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02f0001bc0 con 0x7f02fc138100 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:15.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 1 -- 172.21.15.45:0/1309324875 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f02f0002050 con 0x7f02fc138100 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 4 set_mon_vals no callback set 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f02faffd640 10 monclient: dump: 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.836 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 10 monclient: shutdown 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 -- 172.21.15.45:0/1309324875 >> v1:172.21.15.45:6789/0 conn(0x7f02fc138100 legacy=0x7f02fc13cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 -- 172.21.15.45:0/1309324875 shutdown_connections 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 -- 172.21.15.45:0/1309324875 >> 172.21.15.45:0/1309324875 conn(0x7f02fc0aa110 msgr2=0x7f02fc0aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 -- 172.21.15.45:0/1309324875 shutdown_connections 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.818+0000 7f03025d7640 1 -- 172.21.15.45:0/1309324875 wait complete. 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient: build_initial_monmap 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient: monmap: 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:15.837 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 librados: starting msgr at 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 librados: starting objecter 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 Processor -- start 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 -- start start 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 librados: setting wanted keys 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 librados: calling monclient init 2024-09-16T03:20:15.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient: init 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient(hunting): picked mon.noname-c con 0x7f02fc137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient(hunting): picked mon.noname-a con 0x7f02fc138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient(hunting): picked mon.noname-b con 0x7f02fc13e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02fc1389c0 con 0x7f02fc138100 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02fc138ca0 con 0x7f02fc13e2a0 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02fc1cdaf0 con 0x7f02fc137d20 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient(hunting): authenticate will time out at 1906.473377s 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02fbfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f02fc137d20 0x7f02fc1cccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43408/0 (socket says 172.21.15.45:43408) 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02fbfff640 1 -- 172.21.15.45:0/1070981613 learned_addr learned my addr 172.21.15.45:0/1070981613 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 client.?.objecter ms_handle_connect 0x7f02fc137d20 2024-09-16T03:20:15.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 client.?.objecter ms_handle_connect 0x7f02fc138100 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951767814 0 0) 0x7f02fc1cdaf0 con 0x7f02fc137d20 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): my global_id is 144374 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02e8004ec0 con 0x7f02fc137d20 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 client.?.objecter ms_handle_connect 0x7f02fc13e2a0 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531661837 0 0) 0x7f02fc1389c0 con 0x7f02fc138100 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): my global_id is 134472 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02fc1cdaf0 con 0x7f02fc138100 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 818028831 0 0) 0x7f02fc138ca0 con 0x7f02fc13e2a0 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): my global_id is 164155 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02fc1389c0 con 0x7f02fc13e2a0 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1166050295 0 0) 0x7f02fc1cdaf0 con 0x7f02fc138100 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02fc138ca0 con 0x7f02fc138100 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f02f0001f10 con 0x7f02fc138100 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:20:15.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): dump: 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2567164163 0 0) 0x7f02e8004ec0 con 0x7f02fc137d20 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02fc1cdaf0 con 0x7f02fc137d20 2024-09-16T03:20:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f02ec0019a0 con 0x7f02fc137d20 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): dump: 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.843 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2230578083 0 0) 0x7f02fc138ca0 con 0x7f02fc138100 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 >> v1:172.21.15.45:6790/0 conn(0x7f02fc137d20 legacy=0x7f02fc1cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 >> v1:172.21.15.46:6789/0 conn(0x7f02fc13e2a0 legacy=0x7f02fc1d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 monclient: found mon.a 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc141b10 con 0x7f02fc138100 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient: _finish_auth 0 2024-09-16T03:20:15.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 5 monclient: authenticate success, global_id 134472 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient: _renew_subs 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc0a3160 con 0x7f02fc138100 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 client.134472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient: _renew_subs 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f02fc0a3430 con 0x7f02fc138100 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 librados: init done 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02f00020b0 con 0x7f02fc138100 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f02f0004f80 con 0x7f02fc138100 2024-09-16T03:20:15.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02fa7fc640 4 set_mon_vals no callback set 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient: dump: 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f02dc0065d0 con 0x7f02dc002870 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f02f0010ff0 con 0x7f02fc138100 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 client.134472.objecter ms_handle_connect 0x7f02dc002870 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 client.134472.objecter resend_mon_ops 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(630..630 src has 99..630) ==== 5917+0+0 (unknown 920938494 0 0) 0x7f02f004bb70 con 0x7f02fc138100 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 client.134472.objecter ms_dispatch 0x7f02fc000ba0 osd_map(630..630 src has 99..630) 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 3 client.134472.objecter handle_osd_map got epochs [630,630] > 0 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 3 client.134472.objecter handle_osd_map decoding full epoch 630 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 20 client.134472.objecter dump_active .. 0 homeless 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2235177717 0 0) 0x7f02dc0065d0 con 0x7f02dc002870 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.822+0000 7f02f8ff9640 10 monclient(hunting): my global_id is 164158 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02fc138ca0 con 0x7f02dc002870 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f0300b74640 10 monclient: get_auth_request con 0x7f02e803e200 auth_method 0 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3807081900 0 0) 0x7f02fc138ca0 con 0x7f02dc002870 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02dc0065d0 con 0x7f02dc002870 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f02ec003af0 con 0x7f02dc002870 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 10 monclient: dump: 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.848 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 10 client.134472.objecter ms_handle_connect 0x7f02e803e200 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2975922240 0 0) 0x7f02dc0065d0 con 0x7f02dc002870 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02dc00a170 con 0x7f02dc002870 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f02ec0057d0 con 0x7f02dc002870 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 >> v1:172.21.15.46:6789/0 conn(0x7f02dc002870 legacy=0x7f02dc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.826+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 >> v1:172.21.15.46:6789/0 conn(0x7f02dc002870 legacy=0x7f02dc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:15.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.830+0000 7f03025d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:20:15.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.830+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f02dc00c500 con 0x7f02dc009120 2024-09-16T03:20:15.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 client.134472.objecter ms_handle_connect 0x7f02dc009120 2024-09-16T03:20:15.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 client.134472.objecter resend_mon_ops 2024-09-16T03:20:15.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2426468136 0 0) 0x7f02dc00c500 con 0x7f02dc009120 2024-09-16T03:20:15.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 monclient(hunting): my global_id is 164161 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02dc0065d0 con 0x7f02dc009120 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 542979569 0 0) 0x7f02dc0065d0 con 0x7f02dc009120 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02dc00c500 con 0x7f02dc009120 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f02ec0040a0 con 0x7f02dc009120 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 monclient: dump: 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.852 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2537278430 0 0) 0x7f02dc00c500 con 0x7f02dc009120 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f02dc009cf0 con 0x7f02dc009120 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3967 (unknown 0 0 3930597491) 0x7f02ec003e40 con 0x7f02dc009120 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 >> v1:172.21.15.46:6789/0 conn(0x7f02dc009120 legacy=0x7f02dc009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f02f8ff9640 1 -- 172.21.15.45:0/1070981613 >> v1:172.21.15.46:6789/0 conn(0x7f02dc009120 legacy=0x7f02dc009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:15.853 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 62, 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 18143, 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 3, 2024-09-16T03:20:15.854 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:15.855 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:15.856 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:20:15.857 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:15.861 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:15.863 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:20:15.864 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 10 librados: watch_flush enter 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 10 librados: watch_flush exit 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 20 client.134472.objecter shutdown clearing up homeless session... 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 10 client.134472.objecter successfully canceled tick 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 >> v1:172.21.15.46:6812/149360280 conn(0x7f02e803e200 legacy=0x7f02e80406c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 10 monclient: shutdown 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 >> v1:172.21.15.45:6789/0 conn(0x7f02fc138100 legacy=0x7f02fc1cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f0300b74640 1 -- 172.21.15.45:0/1070981613 reap_dead start 2024-09-16T03:20:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f0300b74640 1 -- 172.21.15.45:0/1070981613 reap_dead start 2024-09-16T03:20:15.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 shutdown_connections 2024-09-16T03:20:15.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 >> 172.21.15.45:0/1070981613 conn(0x7f02fc0aa110 msgr2=0x7f02fc0731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:15.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 shutdown_connections 2024-09-16T03:20:15.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 1 -- 172.21.15.45:0/1070981613 wait complete. 2024-09-16T03:20:15.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.834+0000 7f03025d7640 1 librados: shutdown 2024-09-16T03:20:15.867 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:20:15.867 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T03:20:15.867 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:20:15.867 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T03:20:15.868 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-16T03:20:15.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-16T03:20:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient: get_monmap_and_config 2024-09-16T03:20:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient: build_initial_monmap 2024-09-16T03:20:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient: monmap: 2024-09-16T03:20:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:15.990039+0000 2024-09-16T03:20:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:15.990039+0000 2024-09-16T03:20:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:15.989 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 1 Processor -- start 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 1 -- start start 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient: init 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient(hunting): picked mon.noname-b con 0x7f0b88135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient(hunting): picked mon.noname-a con 0x7f0b8809fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient(hunting): picked mon.noname-c con 0x7f0b8813c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b8809f7e0 con 0x7f0b8809fc40 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b8809b160 con 0x7f0b8813c010 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b8809b440 con 0x7f0b88135950 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:15.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.986+0000 7f0b8e021640 10 monclient(hunting): authenticate will time out at 1906.639928s 2024-09-16T03:20:15.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b86ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0b8809fc40 0x7f0b8813a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37978/0 (socket says 172.21.15.45:37978) 2024-09-16T03:20:15.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b877fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0b88135950 0x7f0b88135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43412/0 (socket says 172.21.15.45:43412) 2024-09-16T03:20:15.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b86ffd640 1 -- 172.21.15.45:0/2099972398 learned_addr learned my addr 172.21.15.45:0/2099972398 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:15.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 -- 172.21.15.45:0/2099972398 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942820173 0 0) 0x7f0b8809f7e0 con 0x7f0b8809fc40 2024-09-16T03:20:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient(hunting): _negotiate 2024-09-16T03:20:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient(hunting): my global_id is 134475 2024-09-16T03:20:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 -- 172.21.15.45:0/2099972398 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b70003530 con 0x7f0b8809fc40 2024-09-16T03:20:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 -- 172.21.15.45:0/2099972398 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2230675121 0 0) 0x7f0b70003530 con 0x7f0b8809fc40 2024-09-16T03:20:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 -- 172.21.15.45:0/2099972398 >> v1:172.21.15.45:6790/0 conn(0x7f0b88135950 legacy=0x7f0b88135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 -- 172.21.15.45:0/2099972398 >> v1:172.21.15.46:6789/0 conn(0x7f0b8813c010 legacy=0x7f0b8813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: _finish_hunting 0 2024-09-16T03:20:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 monclient: found mon.noname-a 2024-09-16T03:20:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 -- 172.21.15.45:0/2099972398 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b88136440 con 0x7f0b8809fc40 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: _finish_auth 0 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: _check_auth_tickets 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 -- 172.21.15.45:0/2099972398 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0b7c0016a0 con 0x7f0b8809fc40 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: dump: 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 -- 172.21.15.45:0/2099972398 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b7c0018c0 con 0x7f0b8809fc40 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 1 -- 172.21.15.45:0/2099972398 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0b7c001d50 con 0x7f0b8809fc40 2024-09-16T03:20:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.990+0000 7f0b867fc640 10 monclient: dump: 2024-09-16T03:20:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:15.998 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:15.998 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:15.998 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:15.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 5 monclient: authenticate success, global_id 134475 2024-09-16T03:20:15.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:15.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 4 set_mon_vals no callback set 2024-09-16T03:20:15.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 10 monclient: shutdown 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 1 -- 172.21.15.45:0/2099972398 >> v1:172.21.15.45:6789/0 conn(0x7f0b8809fc40 legacy=0x7f0b8813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 1 -- 172.21.15.45:0/2099972398 shutdown_connections 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 1 -- 172.21.15.45:0/2099972398 >> 172.21.15.45:0/2099972398 conn(0x7f0b880a2bd0 msgr2=0x7f0b880a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 1 -- 172.21.15.45:0/2099972398 shutdown_connections 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.994+0000 7f0b8e021640 1 -- 172.21.15.45:0/2099972398 wait complete. 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient: build_initial_monmap 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient: monmap: 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 1 librados: starting msgr at 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 1 librados: starting objecter 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 1 Processor -- start 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 1 -- start start 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 1 librados: setting wanted keys 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 1 librados: calling monclient init 2024-09-16T03:20:16.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient: init 2024-09-16T03:20:16.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:16.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:20:16.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient(hunting): picked mon.noname-a con 0x7f0b88135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:16.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient(hunting): picked mon.noname-b con 0x7f0b8813c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:16.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient(hunting): picked mon.noname-c con 0x7f0b881d2ed0 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:16.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b88071110 con 0x7f0b88135950 2024-09-16T03:20:16.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b87fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0b881d2ed0 0x7f0b881d52c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43416/0 (socket says 172.21.15.45:43416) 2024-09-16T03:20:16.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b87fff640 1 -- 172.21.15.45:0/3989669473 learned_addr learned my addr 172.21.15.45:0/3989669473 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:16.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b88072700 con 0x7f0b8813c010 2024-09-16T03:20:16.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b880702a0 con 0x7f0b881d2ed0 2024-09-16T03:20:16.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:16.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b8e021640 10 monclient(hunting): authenticate will time out at 1906.650095s 2024-09-16T03:20:16.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 client.?.objecter ms_handle_connect 0x7f0b881d2ed0 2024-09-16T03:20:16.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:16.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 client.?.objecter ms_handle_connect 0x7f0b88135950 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4066376974 0 0) 0x7f0b88071110 con 0x7f0b88135950 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): my global_id is 134478 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b74004ef0 con 0x7f0b88135950 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420601622 0 0) 0x7f0b880702a0 con 0x7f0b881d2ed0 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:16.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): my global_id is 144380 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b88071110 con 0x7f0b881d2ed0 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2580376628 0 0) 0x7f0b88071110 con 0x7f0b881d2ed0 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b880702a0 con 0x7f0b881d2ed0 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0b80001ca0 con 0x7f0b881d2ed0 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): dump: 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:15.998+0000 7f0b6ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 119615488 0 0) 0x7f0b74004ef0 con 0x7f0b88135950 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b88071110 con 0x7f0b88135950 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0b78001850 con 0x7f0b88135950 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient(hunting): dump: 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3077200105 0 0) 0x7f0b88071110 con 0x7f0b88135950 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 >> v1:172.21.15.45:6790/0 conn(0x7f0b881d2ed0 legacy=0x7f0b881d52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 >> v1:172.21.15.46:6789/0 conn(0x7f0b8813c010 legacy=0x7f0b8806fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 monclient: found mon.a 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:16.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b88072cc0 con 0x7f0b88135950 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: _finish_auth 0 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 5 monclient: authenticate success, global_id 134478 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b780047f0 con 0x7f0b88135950 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 10 monclient: _renew_subs 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0b88071850 con 0x7f0b88135950 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0b78004b40 con 0x7f0b88135950 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:16.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: dump: 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b85ffb640 4 set_mon_vals no callback set 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 10 client.134478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:16.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 10 monclient: _renew_subs 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0b88073700 con 0x7f0b88135950 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 1 librados: init done 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0b88072af0 con 0x7f0b881386b0 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 client.134478.objecter ms_handle_connect 0x7f0b881386b0 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 client.134478.objecter resend_mon_ops 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 268674579 0 0) 0x7f0b88072af0 con 0x7f0b881386b0 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient(hunting): my global_id is 134481 2024-09-16T03:20:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b88071110 con 0x7f0b881386b0 2024-09-16T03:20:16.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f0b78010bb0 con 0x7f0b88135950 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(631..631 src has 99..631) ==== 5930+0+0 (unknown 480103847 0 0) 0x7f0b7804b900 con 0x7f0b88135950 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 client.134478.objecter ms_dispatch 0x7f0b88000ba0 osd_map(631..631 src has 99..631) 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 3 client.134478.objecter handle_osd_map got epochs [631,631] > 0 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 3 client.134478.objecter handle_osd_map decoding full epoch 631 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 20 client.134478.objecter dump_active .. 0 homeless 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 723420925 0 0) 0x7f0b88071110 con 0x7f0b881386b0 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b88072af0 con 0x7f0b881386b0 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0b7c001770 con 0x7f0b881386b0 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.002+0000 7f0b6ffff640 10 monclient: dump: 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:16.014 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.006+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 630712222 0 0) 0x7f0b88072af0 con 0x7f0b881386b0 2024-09-16T03:20:16.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.006+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b880008e0 con 0x7f0b881386b0 2024-09-16T03:20:16.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.006+0000 7f0b87fff640 10 monclient: get_auth_request con 0x7f0b7403e120 auth_method 0 2024-09-16T03:20:16.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.006+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0b7c000ff0 con 0x7f0b881386b0 2024-09-16T03:20:16.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.006+0000 7f0b6ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:16.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.006+0000 7f0b6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:16.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.006+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 >> v1:172.21.15.45:6789/0 conn(0x7f0b881386b0 legacy=0x7f0b88138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.006+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 >> v1:172.21.15.45:6789/0 conn(0x7f0b881386b0 legacy=0x7f0b88138a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:16.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.006+0000 7f0b6ffff640 10 client.134478.objecter ms_handle_connect 0x7f0b7403e120 2024-09-16T03:20:16.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b8e021640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0b8806c3e0 con 0x7f0b881391a0 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 client.134478.objecter ms_handle_connect 0x7f0b881391a0 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 client.134478.objecter resend_mon_ops 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147286807 0 0) 0x7f0b8806c3e0 con 0x7f0b881391a0 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 monclient(hunting): my global_id is 134484 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b88072af0 con 0x7f0b881391a0 2024-09-16T03:20:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2178051043 0 0) 0x7f0b88072af0 con 0x7f0b881391a0 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b8806c3e0 con 0x7f0b881391a0 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f0b7c001d10 con 0x7f0b881391a0 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 monclient: dump: 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:16.019 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3645058598 0 0) 0x7f0b8806c3e0 con 0x7f0b881391a0 2024-09-16T03:20:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f0b881395a0 con 0x7f0b881391a0 2024-09-16T03:20:16.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (unknown 0 0 2946029456) 0x7f0b7c004a90 con 0x7f0b881391a0 2024-09-16T03:20:16.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T03:20:16.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:16.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 >> v1:172.21.15.45:6789/0 conn(0x7f0b881391a0 legacy=0x7f0b881daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b6ffff640 1 -- 172.21.15.45:0/3989669473 >> v1:172.21.15.45:6789/0 conn(0x7f0b881391a0 legacy=0x7f0b881daa00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:16.021 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:20:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: "ret": 0, 2024-09-16T03:20:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T03:20:16.021 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:20:16.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b8e021640 10 librados: watch_flush enter 2024-09-16T03:20:16.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b8e021640 10 librados: watch_flush exit 2024-09-16T03:20:16.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b8e021640 20 client.134478.objecter shutdown clearing up homeless session... 2024-09-16T03:20:16.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.014+0000 7f0b8e021640 10 client.134478.objecter successfully canceled tick 2024-09-16T03:20:16.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 >> v1:172.21.15.46:6812/149360280 conn(0x7f0b7403e120 legacy=0x7f0b740405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b8e021640 10 monclient: shutdown 2024-09-16T03:20:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b8e021640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 >> v1:172.21.15.45:6789/0 conn(0x7f0b88135950 legacy=0x7f0b8806f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b87fff640 1 -- 172.21.15.45:0/3989669473 reap_dead start 2024-09-16T03:20:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b87fff640 1 -- 172.21.15.45:0/3989669473 reap_dead start 2024-09-16T03:20:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 shutdown_connections 2024-09-16T03:20:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 >> 172.21.15.45:0/3989669473 conn(0x7f0b880a2bd0 msgr2=0x7f0b880a1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:16.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 shutdown_connections 2024-09-16T03:20:16.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b8e021640 1 -- 172.21.15.45:0/3989669473 wait complete. 2024-09-16T03:20:16.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.018+0000 7f0b8e021640 1 librados: shutdown 2024-09-16T03:20:16.034 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:20:16.034 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:20:16.035 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:20:16.051 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:20:16.051 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:20:16.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.202+0000 7f775b531640 10 monclient: get_monmap_and_config 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.202+0000 7f775b531640 10 monclient: build_initial_monmap 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.202+0000 7f775b531640 10 monclient: monmap: 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:16.207249+0000 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:16.207249+0000 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:16.206 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:16.206 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:16.206 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.202+0000 7f775b531640 1 Processor -- start 2024-09-16T03:20:16.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.202+0000 7f775b531640 1 -- start start 2024-09-16T03:20:16.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.202+0000 7f775b531640 10 monclient: init 2024-09-16T03:20:16.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:16.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:20:16.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): picked mon.noname-a con 0x7f775409b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): picked mon.noname-b con 0x7f775409b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): picked mon.noname-c con 0x7f775413b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775409cc20 con 0x7f775409b8e0 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775409d820 con 0x7f775413b000 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775409b6a0 con 0x7f775409b250 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): authenticate will time out at 1906.856571s 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f77592cd640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f775409b8e0 legacy=0x7f775409bcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f7758acc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f775409b250 0x7f77541398e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43436/0 (socket says 172.21.15.45:43436) 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f7758acc640 1 -- 172.21.15.45:0/2381483557 learned_addr learned my addr 172.21.15.45:0/2381483557 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 -- 172.21.15.45:0/2381483557 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2285204802 0 0) 0x7f775409b6a0 con 0x7f775409b250 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient(hunting): my global_id is 144383 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 -- 172.21.15.45:0/2381483557 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7740005740 con 0x7f775409b250 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 -- 172.21.15.45:0/2381483557 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3949225845 0 0) 0x7f7740005740 con 0x7f775409b250 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 -- 172.21.15.45:0/2381483557 >> v1:172.21.15.46:6789/0 conn(0x7f775413b000 legacy=0x7f775413d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 -- 172.21.15.45:0/2381483557 >> v1:172.21.15.45:6789/0 conn(0x7f775409b8e0 legacy=0x7f775409bcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 monclient: found mon.noname-b 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 -- 172.21.15.45:0/2381483557 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77540a21f0 con 0x7f775409b250 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: _finish_auth 0 2024-09-16T03:20:16.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 -- 172.21.15.45:0/2381483557 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f77500016a0 con 0x7f775409b250 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 5 monclient: authenticate success, global_id 144383 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: dump: 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.211 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 -- 172.21.15.45:0/2381483557 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77500018c0 con 0x7f775409b250 2024-09-16T03:20:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 1 -- 172.21.15.45:0/2381483557 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7750001cd0 con 0x7f775409b250 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f774bfff640 10 monclient: dump: 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:16.213 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 4 set_mon_vals no callback set 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient: shutdown 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- 172.21.15.45:0/2381483557 >> v1:172.21.15.45:6790/0 conn(0x7f775409b250 legacy=0x7f77541398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- 172.21.15.45:0/2381483557 shutdown_connections 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- 172.21.15.45:0/2381483557 >> 172.21.15.45:0/2381483557 conn(0x7f77540a2bd0 msgr2=0x7f77540a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- 172.21.15.45:0/2381483557 shutdown_connections 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- 172.21.15.45:0/2381483557 wait complete. 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient: build_initial_monmap 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient: monmap: 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:16.214 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:16.215 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:16.215 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:16.215 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:16.215 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.215 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:16.215 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:16.215 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:16.215 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 librados: starting msgr at 2024-09-16T03:20:16.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 librados: starting objecter 2024-09-16T03:20:16.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 Processor -- start 2024-09-16T03:20:16.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- start start 2024-09-16T03:20:16.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 librados: setting wanted keys 2024-09-16T03:20:16.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 librados: calling monclient init 2024-09-16T03:20:16.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient: init 2024-09-16T03:20:16.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): picked mon.noname-b con 0x7f775409b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): picked mon.noname-a con 0x7f775409b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): picked mon.noname-c con 0x7f775413b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77540738f0 con 0x7f775409b8e0 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7754097be0 con 0x7f775409b250 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7754094be0 con 0x7f775413b000 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f7758acc640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f775409b8e0 legacy=0x7f77540944d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f775b531640 10 monclient(hunting): authenticate will time out at 1906.859853s 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f7759ace640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f775413b000 0x7f77541d5210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43438/0 (socket says 172.21.15.45:43438) 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.206+0000 7f7759ace640 1 -- 172.21.15.45:0/1885379261 learned_addr learned my addr 172.21.15.45:0/1885379261 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 client.?.objecter ms_handle_connect 0x7f775413b000 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3357032059 0 0) 0x7f7754094be0 con 0x7f775413b000 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): my global_id is 144386 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7738004ec0 con 0x7f775413b000 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 client.?.objecter ms_handle_connect 0x7f775409b250 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:16.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 736600075 0 0) 0x7f7738004ec0 con 0x7f775413b000 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7754094be0 con 0x7f775413b000 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f774c001d40 con 0x7f775413b000 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): dump: 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:16.222 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975867425 0 0) 0x7f7754097be0 con 0x7f775409b250 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient(hunting): my global_id is 164167 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7738004ec0 con 0x7f775409b250 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3321889782 0 0) 0x7f7754094be0 con 0x7f775413b000 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 >> v1:172.21.15.46:6789/0 conn(0x7f775409b250 legacy=0x7f7754093dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 >> v1:172.21.15.45:6789/0 conn(0x7f775409b8e0 legacy=0x7f77540944d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:20:16.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 monclient: found mon.c 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77540a5b40 con 0x7f775413b000 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient: _finish_auth 0 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 5 monclient: authenticate success, global_id 144386 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 10 monclient: _renew_subs 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f775409d2c0 con 0x7f775413b000 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 10 client.144386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 10 monclient: _renew_subs 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f77540963e0 con 0x7f775413b000 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 1 librados: init done 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f774c002000 con 0x7f775413b000 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f774c003df0 con 0x7f775413b000 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f774b7fe640 4 set_mon_vals no callback set 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 monclient: dump: 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:16.225 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7734005090 con 0x7f775413b000 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f774c003f90 con 0x7f775413b000 2024-09-16T03:20:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(631..631 src has 99..631) ==== 5930+0+0 (unknown 480103847 0 0) 0x7f774c04bd00 con 0x7f775413b000 2024-09-16T03:20:16.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 client.144386.objecter ms_dispatch 0x7f7754000ba0 osd_map(631..631 src has 99..631) 2024-09-16T03:20:16.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 3 client.144386.objecter handle_osd_map got epochs [631,631] > 0 2024-09-16T03:20:16.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 3 client.144386.objecter handle_osd_map decoding full epoch 631 2024-09-16T03:20:16.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 20 client.144386.objecter dump_active .. 0 homeless 2024-09-16T03:20:16.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f77592cd640 10 monclient: get_auth_request con 0x7f773803e090 auth_method 0 2024-09-16T03:20:16.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.210+0000 7f7749ffb640 10 client.144386.objecter ms_handle_connect 0x7f773803e090 2024-09-16T03:20:16.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.214+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f774c016770 con 0x7f775413b000 2024-09-16T03:20:16.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.214+0000 7f7749ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:16.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.214+0000 7f7749ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:16.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.450+0000 7f775b531640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:20:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.450+0000 7f775b531640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.450+0000 7f775b531640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.450+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7734005380 con 0x7f775413b000 2024-09-16T03:20:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.450+0000 7f7749ffb640 1 -- 172.21.15.45:0/1885379261 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2196287052) 0x7f774c020020 con 0x7f775413b000 2024-09-16T03:20:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.450+0000 7f7749ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.450+0000 7f7749ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:16.453 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:20:16.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 10 librados: watch_flush enter 2024-09-16T03:20:16.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 10 librados: watch_flush exit 2024-09-16T03:20:16.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 20 client.144386.objecter shutdown clearing up homeless session... 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 10 client.144386.objecter successfully canceled tick 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 >> v1:172.21.15.46:6812/149360280 conn(0x7f773803e090 legacy=0x7f7738040550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 10 monclient: shutdown 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 >> v1:172.21.15.45:6790/0 conn(0x7f775413b000 legacy=0x7f77541d5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 shutdown_connections 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 >> 172.21.15.45:0/1885379261 conn(0x7f77540a2bd0 msgr2=0x7f775408df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 shutdown_connections 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 1 -- 172.21.15.45:0/1885379261 wait complete. 2024-09-16T03:20:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:16.454+0000 7f775b531640 1 librados: shutdown 2024-09-16T03:20:19.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:20:19.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient: get_monmap_and_config 2024-09-16T03:20:19.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient: build_initial_monmap 2024-09-16T03:20:19.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient: monmap: 2024-09-16T03:20:19.600 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:19.600 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:19.600 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:19.602375+0000 2024-09-16T03:20:19.600 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:19.602375+0000 2024-09-16T03:20:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:19.601 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 1 Processor -- start 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 1 -- start start 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient: init 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient(hunting): picked mon.noname-a con 0x7f3bc8135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient(hunting): picked mon.noname-b con 0x7f3bc809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient(hunting): picked mon.noname-c con 0x7f3bc813c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc809f7e0 con 0x7f3bc8135950 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bc77fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f3bc8135950 legacy=0x7f3bc8135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc809b160 con 0x7f3bc813c010 2024-09-16T03:20:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc809b440 con 0x7f3bc809fc40 2024-09-16T03:20:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bce12b640 10 monclient(hunting): authenticate will time out at 1910.251882s 2024-09-16T03:20:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bc6ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3bc809fc40 0x7f3bc813a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43444/0 (socket says 172.21.15.45:43444) 2024-09-16T03:20:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.598+0000 7f3bc6ffd640 1 -- 172.21.15.45:0/3410975947 learned_addr learned my addr 172.21.15.45:0/3410975947 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 -- 172.21.15.45:0/3410975947 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3802518421 0 0) 0x7f3bc809b440 con 0x7f3bc809fc40 2024-09-16T03:20:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient(hunting): my global_id is 144389 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 -- 172.21.15.45:0/3410975947 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bb0003530 con 0x7f3bc809fc40 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 -- 172.21.15.45:0/3410975947 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1662062192 0 0) 0x7f3bb0003530 con 0x7f3bc809fc40 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 -- 172.21.15.45:0/3410975947 >> v1:172.21.15.46:6789/0 conn(0x7f3bc813c010 legacy=0x7f3bc813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 -- 172.21.15.45:0/3410975947 >> v1:172.21.15.45:6789/0 conn(0x7f3bc8135950 legacy=0x7f3bc8135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: _finish_hunting 0 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 monclient: found mon.noname-b 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 -- 172.21.15.45:0/3410975947 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc8136440 con 0x7f3bc809fc40 2024-09-16T03:20:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: _finish_auth 0 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 -- 172.21.15.45:0/3410975947 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3bbc0016a0 con 0x7f3bc809fc40 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: dump: 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 5 monclient: authenticate success, global_id 144389 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 -- 172.21.15.45:0/3410975947 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3bbc0018c0 con 0x7f3bc809fc40 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 1 -- 172.21.15.45:0/3410975947 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3bbc001d50 con 0x7f3bc809fc40 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 4 set_mon_vals no callback set 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc67fc640 10 monclient: dump: 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: shutdown 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- 172.21.15.45:0/3410975947 >> v1:172.21.15.45:6790/0 conn(0x7f3bc809fc40 legacy=0x7f3bc813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- 172.21.15.45:0/3410975947 shutdown_connections 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- 172.21.15.45:0/3410975947 >> 172.21.15.45:0/3410975947 conn(0x7f3bc80a2bd0 msgr2=0x7f3bc80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- 172.21.15.45:0/3410975947 shutdown_connections 2024-09-16T03:20:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- 172.21.15.45:0/3410975947 wait complete. 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: build_initial_monmap 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: monmap: 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 librados: starting msgr at 2024-09-16T03:20:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 librados: starting objecter 2024-09-16T03:20:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 Processor -- start 2024-09-16T03:20:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- start start 2024-09-16T03:20:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 librados: setting wanted keys 2024-09-16T03:20:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 librados: calling monclient init 2024-09-16T03:20:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: init 2024-09-16T03:20:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:20:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient(hunting): picked mon.noname-c con 0x7f3bc809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient(hunting): picked mon.noname-b con 0x7f3bc8135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient(hunting): picked mon.noname-a con 0x7f3bc813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc80714d0 con 0x7f3bc813c010 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc80717b0 con 0x7f3bc8135950 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc7fff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f3bc813c010 legacy=0x7f3bc806f1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc77fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3bc809fc40 0x7f3bc806e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43456/0 (socket says 172.21.15.45:43456) 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc8070290 con 0x7f3bc809fc40 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bc77fe640 1 -- 172.21.15.45:0/3027999816 learned_addr learned my addr 172.21.15.45:0/3027999816 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient(hunting): authenticate will time out at 1910.255051s 2024-09-16T03:20:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 client.?.objecter ms_handle_connect 0x7f3bc809fc40 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1986133859 0 0) 0x7f3bc8070290 con 0x7f3bc809fc40 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient(hunting): my global_id is 144392 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bb4004ec0 con 0x7f3bc809fc40 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 client.?.objecter ms_handle_connect 0x7f3bc8135950 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3557955309 0 0) 0x7f3bb4004ec0 con 0x7f3bc809fc40 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3bc8070290 con 0x7f3bc809fc40 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3bb80041e0 con 0x7f3bc809fc40 2024-09-16T03:20:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient(hunting): dump: 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 738516403 0 0) 0x7f3bc8070290 con 0x7f3bc809fc40 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 >> v1:172.21.15.46:6789/0 conn(0x7f3bc8135950 legacy=0x7f3bc806eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 >> v1:172.21.15.45:6789/0 conn(0x7f3bc813c010 legacy=0x7f3bc806f1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient: _finish_hunting 0 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 monclient: found mon.c 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc806f9d0 con 0x7f3bc809fc40 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient: _finish_auth 0 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 10 monclient: _check_auth_tickets 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3baffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 5 monclient: authenticate success, global_id 144392 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: _renew_subs 2024-09-16T03:20:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc80a1040 con 0x7f3bc809fc40 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 client.144392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: _renew_subs 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3bc81d3b00 con 0x7f3bc809fc40 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.602+0000 7f3bce12b640 1 librados: init done 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3bb8004400 con 0x7f3bc809fc40 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3bc5ffb640 4 set_mon_vals no callback set 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3bb8004850 con 0x7f3bc809fc40 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 10 monclient: dump: 2024-09-16T03:20:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3bce12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3bce12b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3bce12b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba8005090 con 0x7f3bc809fc40 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f3bb80049f0 con 0x7f3bc809fc40 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(631..631 src has 99..631) ==== 5930+0+0 (unknown 480103847 0 0) 0x7f3bb804bae0 con 0x7f3bc809fc40 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 10 client.144392.objecter ms_dispatch 0x7f3bc8000ba0 osd_map(631..631 src has 99..631) 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 3 client.144392.objecter handle_osd_map got epochs [631,631] > 0 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 3 client.144392.objecter handle_osd_map decoding full epoch 631 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 20 client.144392.objecter dump_active .. 0 homeless 2024-09-16T03:20:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3bc6ffd640 10 monclient: get_auth_request con 0x7f3bb403de60 auth_method 0 2024-09-16T03:20:19.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.606+0000 7f3baffff640 10 client.144392.objecter ms_handle_connect 0x7f3bb403de60 2024-09-16T03:20:19.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.610+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f3bb8016c00 con 0x7f3bc809fc40 2024-09-16T03:20:19.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.610+0000 7f3baffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:19.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.610+0000 7f3baffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.846+0000 7f3bce12b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:20:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.846+0000 7f3bce12b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.846+0000 7f3bce12b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:19.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.846+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3ba8005380 con 0x7f3bc809fc40 2024-09-16T03:20:19.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.846+0000 7f3baffff640 1 -- 172.21.15.45:0/3027999816 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 1303463907) 0x7f3bb8016500 con 0x7f3bc809fc40 2024-09-16T03:20:19.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.846+0000 7f3baffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:19.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.846+0000 7f3baffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:20:19.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 10 librados: watch_flush enter 2024-09-16T03:20:19.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 10 librados: watch_flush exit 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 20 client.144392.objecter shutdown clearing up homeless session... 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 10 client.144392.objecter successfully canceled tick 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 >> v1:172.21.15.46:6812/149360280 conn(0x7f3bb403de60 legacy=0x7f3bb4040320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 10 monclient: shutdown 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 >> v1:172.21.15.45:6790/0 conn(0x7f3bc809fc40 legacy=0x7f3bc806e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 shutdown_connections 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 >> 172.21.15.45:0/3027999816 conn(0x7f3bc80a2bd0 msgr2=0x7f3bc813eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 shutdown_connections 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 1 -- 172.21.15.45:0/3027999816 wait complete. 2024-09-16T03:20:19.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:19.850+0000 7f3bce12b640 1 librados: shutdown 2024-09-16T03:20:22.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:20:23.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:22.998+0000 7f257f8ea640 10 monclient: get_monmap_and_config 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:22.998+0000 7f257f8ea640 10 monclient: build_initial_monmap 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:22.998+0000 7f257f8ea640 10 monclient: monmap: 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:23.002998+0000 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:23.002998+0000 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:23.002 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:22.998+0000 7f257f8ea640 1 Processor -- start 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:22.998+0000 7f257f8ea640 1 -- start start 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:22.998+0000 7f257f8ea640 10 monclient: init 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 10 monclient(hunting): picked mon.noname-a con 0x7f2578135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 10 monclient(hunting): picked mon.noname-c con 0x7f257809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 10 monclient(hunting): picked mon.noname-b con 0x7f257813c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257809f7e0 con 0x7f2578135950 2024-09-16T03:20:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257809b160 con 0x7f257809fc40 2024-09-16T03:20:23.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257809b440 con 0x7f257813c010 2024-09-16T03:20:23.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:23.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257d686640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f2578135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:23.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257f8ea640 10 monclient(hunting): authenticate will time out at 1913.652543s 2024-09-16T03:20:23.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257de87640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f257813c010 0x7f257813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:49504/0 (socket says 172.21.15.45:49504) 2024-09-16T03:20:23.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.002+0000 7f257de87640 1 -- 172.21.15.45:0/2685793105 learned_addr learned my addr 172.21.15.45:0/2685793105 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:23.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.206+0000 7f257d686640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f2578135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:23.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:23.610+0000 7f257d686640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f2578135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:24.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:24.414+0000 7f257d686640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f2578135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:26.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 10 monclient(hunting): tick 2024-09-16T03:20:26.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:20:26.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 1 monclient(hunting): continuing hunt 2024-09-16T03:20:26.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:20:26.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6790/0 conn(0x7f257813c010 legacy=0x7f257813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:26.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.46:6789/0 conn(0x7f257809fc40 legacy=0x7f257813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:26.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f2578135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:20:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:20:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f2564003700 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f257809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f2578135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2578136440 con 0x7f2578135950 2024-09-16T03:20:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257809f7e0 con 0x7f257809fc40 2024-09-16T03:20:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2564010400 con 0x7f2564003700 2024-09-16T03:20:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f2576ffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:26.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.002+0000 7f257de87640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f256400cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.206+0000 7f257de87640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f256400cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:26.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:26.610+0000 7f257de87640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f256400cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:27.414+0000 7f257de87640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f256400cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 10 monclient(hunting): tick 2024-09-16T03:20:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:20:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 1 monclient(hunting): continuing hunt 2024-09-16T03:20:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6790/0 conn(0x7f2564003700 legacy=0x7f2564005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.46:6789/0 conn(0x7f257809fc40 legacy=0x7f25640093e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f2578135950 legacy=0x7f256400cbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f256400f430 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f25640105d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f2564011160 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25640116a0 con 0x7f256400f430 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2578136440 con 0x7f2564011160 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 1 -- 172.21.15.45:0/2685793105 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2564009b10 con 0x7f25640105d0 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f2576ffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.002+0000 7f257d686640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f256400f430 legacy=0x7f256400f870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.206+0000 7f257d686640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f256400f430 legacy=0x7f256400f870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:29.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:29.610+0000 7f257d686640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f256400f430 legacy=0x7f256400f870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:30.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:30.414+0000 7f257d686640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f256400f430 legacy=0x7f256400f870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:31.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.046+0000 7f2577fff640 1 -- 172.21.15.45:0/2685793105 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3349423764 0 0) 0x7f2564009b10 con 0x7f25640105d0 2024-09-16T03:20:31.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.046+0000 7f2577fff640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.046+0000 7f2577fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.046+0000 7f2577fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.046+0000 7f2577fff640 10 monclient(hunting): my global_id is 144395 2024-09-16T03:20:31.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 1 -- 172.21.15.45:0/2685793105 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2560003530 con 0x7f25640105d0 2024-09-16T03:20:31.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 1 -- 172.21.15.45:0/2685793105 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 658396588 0 0) 0x7f2560003530 con 0x7f25640105d0 2024-09-16T03:20:31.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.46:6789/0 conn(0x7f2564011160 legacy=0x7f256400cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6789/0 conn(0x7f256400f430 legacy=0x7f256400f870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:20:31.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: _finish_hunting 0 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 1 monclient: found mon.noname-b 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 1 -- 172.21.15.45:0/2685793105 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2564009d80 con 0x7f25640105d0 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: _finish_auth 0 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: _check_auth_tickets 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 5 monclient: authenticate success, global_id 144395 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 1 -- 172.21.15.45:0/2685793105 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f256c001560 con 0x7f25640105d0 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: dump: 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 1 -- 172.21.15.45:0/2685793105 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f256c001700 con 0x7f25640105d0 2024-09-16T03:20:31.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 4 set_mon_vals no callback set 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 1 -- 172.21.15.45:0/2685793105 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f256c001b90 con 0x7f25640105d0 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f2577fff640 10 monclient: dump: 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.054 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.055 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.055 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.055 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.055 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 10 monclient: shutdown 2024-09-16T03:20:31.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:31.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 1 -- 172.21.15.45:0/2685793105 >> v1:172.21.15.45:6790/0 conn(0x7f25640105d0 legacy=0x7f2564010a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 1 -- 172.21.15.45:0/2685793105 shutdown_connections 2024-09-16T03:20:31.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 1 -- 172.21.15.45:0/2685793105 >> 172.21.15.45:0/2685793105 conn(0x7f25780a2bd0 msgr2=0x7f25780a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 1 -- 172.21.15.45:0/2685793105 shutdown_connections 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.050+0000 7f257f8ea640 1 -- 172.21.15.45:0/2685793105 wait complete. 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient: build_initial_monmap 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient: monmap: 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:31.061 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 librados: starting msgr at 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 librados: starting objecter 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 Processor -- start 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 -- start start 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 librados: setting wanted keys 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 librados: calling monclient init 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient: init 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient(hunting): picked mon.noname-a con 0x7f256400f430 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:31.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient(hunting): picked mon.noname-b con 0x7f25640105d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient(hunting): picked mon.noname-c con 0x7f2564011160 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257809d2c0 con 0x7f256400f430 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257d686640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f256400f430 legacy=0x7f25781ca5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257813a620 con 0x7f25640105d0 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257813ae80 con 0x7f2564011160 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient(hunting): authenticate will time out at 1921.705971s 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257de87640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2564011160 0x7f25781d2970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46510/0 (socket says 172.21.15.45:46510) 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257de87640 1 -- 172.21.15.45:0/254785788 learned_addr learned my addr 172.21.15.45:0/254785788 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 client.?.objecter ms_handle_connect 0x7f2564011160 2024-09-16T03:20:31.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 client.?.objecter ms_handle_connect 0x7f25640105d0 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3453343634 0 0) 0x7f257813ae80 con 0x7f2564011160 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient(hunting): my global_id is 144398 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2564011540 con 0x7f2564011160 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4115666022 0 0) 0x7f2564011540 con 0x7f2564011160 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f257813ae80 con 0x7f2564011160 2024-09-16T03:20:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2570001b30 con 0x7f2564011160 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient(hunting): dump: 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.065 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3715537285 0 0) 0x7f257813ae80 con 0x7f2564011160 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 >> v1:172.21.15.46:6789/0 conn(0x7f25640105d0 legacy=0x7f2578073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 >> v1:172.21.15.45:6789/0 conn(0x7f256400f430 legacy=0x7f25781ca5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 monclient: found mon.c 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25781394d0 con 0x7f2564011160 2024-09-16T03:20:31.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient: _finish_auth 0 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 5 monclient: authenticate success, global_id 144398 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient: _renew_subs 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2578139f20 con 0x7f2564011160 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 client.144398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient: _renew_subs 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2578074230 con 0x7f2564011160 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f257f8ea640 1 librados: init done 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f25700020c0 con 0x7f2564011160 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2570004dd0 con 0x7f2564011160 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f25777fe640 4 set_mon_vals no callback set 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.054+0000 7f2575ffb640 10 monclient: dump: 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.068 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f257f8ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f257f8ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f257f8ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2558005090 con 0x7f2564011160 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f2570004f70 con 0x7f2564011160 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(631..631 src has 99..631) ==== 5930+0+0 (unknown 480103847 0 0) 0x7f257004bbd0 con 0x7f2564011160 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f2575ffb640 10 client.144398.objecter ms_dispatch 0x7f2578000ba0 osd_map(631..631 src has 99..631) 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f2575ffb640 3 client.144398.objecter handle_osd_map got epochs [631,631] > 0 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f2575ffb640 3 client.144398.objecter handle_osd_map decoding full epoch 631 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f2575ffb640 20 client.144398.objecter dump_active .. 0 homeless 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f257d686640 10 monclient: get_auth_request con 0x7f256403f830 auth_method 0 2024-09-16T03:20:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.058+0000 7f2575ffb640 10 client.144398.objecter ms_handle_connect 0x7f256403f830 2024-09-16T03:20:31.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.062+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f2570016640 con 0x7f2564011160 2024-09-16T03:20:31.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.062+0000 7f2575ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:31.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.062+0000 7f2575ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:31.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.306+0000 7f257f8ea640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:20:31.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.306+0000 7f257f8ea640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:31.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.306+0000 7f257f8ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.306+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2558005380 con 0x7f2564011160 2024-09-16T03:20:31.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.306+0000 7f2575ffb640 1 -- 172.21.15.45:0/254785788 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1721003845) 0x7f257001fef0 con 0x7f2564011160 2024-09-16T03:20:31.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.306+0000 7f2575ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:20:31.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.306+0000 7f2575ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:31.309 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":66,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:20:31.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 10 librados: watch_flush enter 2024-09-16T03:20:31.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 10 librados: watch_flush exit 2024-09-16T03:20:31.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 20 client.144398.objecter shutdown clearing up homeless session... 2024-09-16T03:20:31.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 10 client.144398.objecter successfully canceled tick 2024-09-16T03:20:31.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 >> v1:172.21.15.46:6812/149360280 conn(0x7f256403f830 legacy=0x7f2564041cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 10 monclient: shutdown 2024-09-16T03:20:31.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:31.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 >> v1:172.21.15.45:6790/0 conn(0x7f2564011160 legacy=0x7f25781d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 shutdown_connections 2024-09-16T03:20:31.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 >> 172.21.15.45:0/254785788 conn(0x7f25780a2bd0 msgr2=0x7f2578097210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:31.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 shutdown_connections 2024-09-16T03:20:31.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 1 -- 172.21.15.45:0/254785788 wait complete. 2024-09-16T03:20:31.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.310+0000 7f257f8ea640 1 librados: shutdown 2024-09-16T03:20:31.328 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:20:31.329 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:20:31.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:20:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient: get_monmap_and_config 2024-09-16T03:20:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient: build_initial_monmap 2024-09-16T03:20:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient: monmap: 2024-09-16T03:20:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:31.452937+0000 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:31.452937+0000 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 1 Processor -- start 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 1 -- start start 2024-09-16T03:20:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient: init 2024-09-16T03:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient(hunting): picked mon.noname-b con 0x7fe264137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient(hunting): picked mon.noname-c con 0x7fe264138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient(hunting): picked mon.noname-a con 0x7fe26413e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2640a4f90 con 0x7fe26413e2a0 2024-09-16T03:20:31.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2640a4cb0 con 0x7fe264138100 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe26409dc70 con 0x7fe264137d20 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient(hunting): authenticate will time out at 1922.102359s 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe2637fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fe26413e2a0 legacy=0x7fe264140690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe262ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe264137d20 0x7fe2640a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46526/0 (socket says 172.21.15.45:46526) 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe262ffd640 1 -- 172.21.15.45:0/436879797 learned_addr learned my addr 172.21.15.45:0/436879797 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 -- 172.21.15.45:0/436879797 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1773720885 0 0) 0x7fe26409dc70 con 0x7fe264137d20 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient(hunting): my global_id is 144401 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 -- 172.21.15.45:0/436879797 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe24c003530 con 0x7fe264137d20 2024-09-16T03:20:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 -- 172.21.15.45:0/436879797 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 714240554 0 0) 0x7fe24c003530 con 0x7fe264137d20 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 -- 172.21.15.45:0/436879797 >> v1:172.21.15.46:6789/0 conn(0x7fe264138100 legacy=0x7fe26413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 -- 172.21.15.45:0/436879797 >> v1:172.21.15.45:6789/0 conn(0x7fe26413e2a0 legacy=0x7fe264140690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 monclient: found mon.noname-b 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 -- 172.21.15.45:0/436879797 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26409cc50 con 0x7fe264137d20 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: _finish_auth 0 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 5 monclient: authenticate success, global_id 144401 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 -- 172.21.15.45:0/436879797 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe2540016a0 con 0x7fe264137d20 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: dump: 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 -- 172.21.15.45:0/436879797 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2540018c0 con 0x7fe264137d20 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 4 set_mon_vals no callback set 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 1 -- 172.21.15.45:0/436879797 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe254001d50 con 0x7fe264137d20 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe261ffb640 10 monclient: dump: 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 10 monclient: shutdown 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.450+0000 7fe26966e640 1 -- 172.21.15.45:0/436879797 >> v1:172.21.15.45:6790/0 conn(0x7fe264137d20 legacy=0x7fe2640a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- 172.21.15.45:0/436879797 shutdown_connections 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- 172.21.15.45:0/436879797 >> 172.21.15.45:0/436879797 conn(0x7fe2640aa110 msgr2=0x7fe2640aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- 172.21.15.45:0/436879797 shutdown_connections 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- 172.21.15.45:0/436879797 wait complete. 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient: build_initial_monmap 2024-09-16T03:20:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient: monmap: 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 librados: starting msgr at 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 librados: starting objecter 2024-09-16T03:20:31.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 Processor -- start 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- start start 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 librados: setting wanted keys 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 librados: calling monclient init 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient: init 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient(hunting): picked mon.noname-a con 0x7fe264137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient(hunting): picked mon.noname-b con 0x7fe264138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient(hunting): picked mon.noname-c con 0x7fe26413e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2640a2760 con 0x7fe264137d20 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2640a2120 con 0x7fe264138100 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe264141a60 con 0x7fe26413e2a0 2024-09-16T03:20:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient(hunting): authenticate will time out at 1922.105543s 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe262ffd640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fe264137d20 legacy=0x7fe2640a1300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe2637fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe26413e2a0 0x7fe2641d4f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46542/0 (socket says 172.21.15.45:46542) 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe2637fe640 1 -- 172.21.15.45:0/713557003 learned_addr learned my addr 172.21.15.45:0/713557003 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 client.?.objecter ms_handle_connect 0x7fe26413e2a0 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465285372 0 0) 0x7fe264141a60 con 0x7fe26413e2a0 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): my global_id is 144404 2024-09-16T03:20:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe250004ec0 con 0x7fe26413e2a0 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 client.?.objecter ms_handle_connect 0x7fe264138100 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3408339130 0 0) 0x7fe250004ec0 con 0x7fe26413e2a0 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe264141a60 con 0x7fe26413e2a0 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe25c004440 con 0x7fe26413e2a0 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): dump: 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4284232531 0 0) 0x7fe264141a60 con 0x7fe26413e2a0 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 >> v1:172.21.15.46:6789/0 conn(0x7fe264138100 legacy=0x7fe2640a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 >> v1:172.21.15.45:6789/0 conn(0x7fe264137d20 legacy=0x7fe2640a1300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 monclient: found mon.c 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26413db70 con 0x7fe26413e2a0 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 5 monclient: authenticate success, global_id 144404 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient: _renew_subs 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe2640a3fa0 con 0x7fe26413e2a0 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 client.144404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient: _renew_subs 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe264138fc0 con 0x7fe26413e2a0 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 librados: init done 2024-09-16T03:20:31.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe25c0045e0 con 0x7fe26413e2a0 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe25c0049f0 con 0x7fe26413e2a0 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe260ff9640 4 set_mon_vals no callback set 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient: dump: 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe2440065d0 con 0x7fe244002870 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 client.144404.objecter ms_handle_connect 0x7fe244002870 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 client.144404.objecter resend_mon_ops 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fe25c004b90 con 0x7fe26413e2a0 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(632..632 src has 99..632) ==== 5925+0+0 (unknown 723040834 0 0) 0x7fe25c04bc70 con 0x7fe26413e2a0 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 client.144404.objecter ms_dispatch 0x7fe264000ba0 osd_map(632..632 src has 99..632) 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 3 client.144404.objecter handle_osd_map got epochs [632,632] > 0 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 3 client.144404.objecter handle_osd_map decoding full epoch 632 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 20 client.144404.objecter dump_active .. 0 homeless 2024-09-16T03:20:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185444615 0 0) 0x7fe2440065d0 con 0x7fe244002870 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.454+0000 7fe24b7fe640 10 monclient(hunting): my global_id is 144407 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe264141a60 con 0x7fe244002870 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe2627fc640 10 monclient: get_auth_request con 0x7fe25003df20 auth_method 0 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3899254814 0 0) 0x7fe264141a60 con 0x7fe244002870 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2440065d0 con 0x7fe244002870 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe254001500 con 0x7fe244002870 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 10 monclient: dump: 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 971437749 0 0) 0x7fe2440065d0 con 0x7fe244002870 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe24400a170 con 0x7fe244002870 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 10 client.144404.objecter ms_handle_connect 0x7fe25003df20 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe254001ce0 con 0x7fe244002870 2024-09-16T03:20:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:31.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:31.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 >> v1:172.21.15.45:6790/0 conn(0x7fe244002870 legacy=0x7fe244004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.458+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 >> v1:172.21.15.45:6790/0 conn(0x7fe244002870 legacy=0x7fe244004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:31.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe26966e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:20:31.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe244009e60 con 0x7fe244009120 2024-09-16T03:20:31.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 client.144404.objecter ms_handle_connect 0x7fe244009120 2024-09-16T03:20:31.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 client.144404.objecter resend_mon_ops 2024-09-16T03:20:31.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1426632364 0 0) 0x7fe244009e60 con 0x7fe244009120 2024-09-16T03:20:31.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 monclient(hunting): my global_id is 144410 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2440065d0 con 0x7fe244009120 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2834079566 0 0) 0x7fe2440065d0 con 0x7fe244009120 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe244009e60 con 0x7fe244009120 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe254004b70 con 0x7fe244009120 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 monclient: dump: 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2801394395 0 0) 0x7fe244009e60 con 0x7fe244009120 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe24400c400 con 0x7fe244009120 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4124 (unknown 0 0 1581342092) 0x7fe254005fa0 con 0x7fe244009120 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 >> v1:172.21.15.45:6790/0 conn(0x7fe244009120 legacy=0x7fe2440095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.462+0000 7fe24b7fe640 1 -- 172.21.15.45:0/713557003 >> v1:172.21.15.45:6790/0 conn(0x7fe244009120 legacy=0x7fe2440095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 66, 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 256536, 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:20:31.481 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:31.482 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:31.483 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:20:31.484 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:31.485 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:31.486 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:31.487 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:31.489 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:31.491 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:20:31.492 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:20:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 10 librados: watch_flush enter 2024-09-16T03:20:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 10 librados: watch_flush exit 2024-09-16T03:20:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 20 client.144404.objecter shutdown clearing up homeless session... 2024-09-16T03:20:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 10 client.144404.objecter successfully canceled tick 2024-09-16T03:20:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 >> v1:172.21.15.46:6812/149360280 conn(0x7fe25003df20 legacy=0x7fe2500403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 10 monclient: shutdown 2024-09-16T03:20:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 >> v1:172.21.15.45:6790/0 conn(0x7fe26413e2a0 legacy=0x7fe2641d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe2637fe640 1 -- 172.21.15.45:0/713557003 reap_dead start 2024-09-16T03:20:31.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe2637fe640 1 -- 172.21.15.45:0/713557003 reap_dead start 2024-09-16T03:20:31.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 shutdown_connections 2024-09-16T03:20:31.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 >> 172.21.15.45:0/713557003 conn(0x7fe2640aa110 msgr2=0x7fe2640731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:31.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 shutdown_connections 2024-09-16T03:20:31.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 1 -- 172.21.15.45:0/713557003 wait complete. 2024-09-16T03:20:31.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.466+0000 7fe26966e640 1 librados: shutdown 2024-09-16T03:20:31.495 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:20:31.495 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:20:31.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:20:31.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient: get_monmap_and_config 2024-09-16T03:20:31.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient: build_initial_monmap 2024-09-16T03:20:31.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient: monmap: 2024-09-16T03:20:31.611 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:31.612 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.612 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:20:31.613563+0000 2024-09-16T03:20:31.612 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:20:31.613563+0000 2024-09-16T03:20:31.612 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:31.612 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.612 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:31.612 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:20:31.612 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:20:31.612 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 1 Processor -- start 2024-09-16T03:20:31.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 1 -- start start 2024-09-16T03:20:31.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient: init 2024-09-16T03:20:31.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:31.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:20:31.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient(hunting): picked mon.noname-c con 0x7f396409c570 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient(hunting): picked mon.noname-a con 0x7f3964134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient(hunting): picked mon.noname-b con 0x7f39641347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3964135150 con 0x7f3964134320 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3964074470 con 0x7f396409c570 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3964074750 con 0x7f39641347d0 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient(hunting): authenticate will time out at 1922.262994s 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f39695dc640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f3964134320 legacy=0x7f396413cb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396a5de640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f39641347d0 0x7f39641402a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46554/0 (socket says 172.21.15.45:46554) 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396a5de640 1 -- 172.21.15.45:0/1377687584 learned_addr learned my addr 172.21.15.45:0/1377687584 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 -- 172.21.15.45:0/1377687584 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2640989730 0 0) 0x7f3964074750 con 0x7f39641347d0 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient(hunting): my global_id is 144413 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 -- 172.21.15.45:0/1377687584 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3950003530 con 0x7f39641347d0 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 -- 172.21.15.45:0/1377687584 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 82702415 0 0) 0x7f3950003530 con 0x7f39641347d0 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 -- 172.21.15.45:0/1377687584 >> v1:172.21.15.46:6789/0 conn(0x7f396409c570 legacy=0x7f396409c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 -- 172.21.15.45:0/1377687584 >> v1:172.21.15.45:6789/0 conn(0x7f3964134320 legacy=0x7f396413cb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: _finish_hunting 0 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 monclient: found mon.noname-b 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 -- 172.21.15.45:0/1377687584 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3964074a30 con 0x7f39641347d0 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: _finish_auth 0 2024-09-16T03:20:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: _check_auth_tickets 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 -- 172.21.15.45:0/1377687584 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f395c0016a0 con 0x7f39641347d0 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: dump: 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 5 monclient: authenticate success, global_id 144413 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 -- 172.21.15.45:0/1377687584 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f395c001840 con 0x7f39641347d0 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 1 -- 172.21.15.45:0/1377687584 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f395c001c50 con 0x7f39641347d0 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 10 monclient: get_monmap_and_config success 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f396c041640 4 set_mon_vals no callback set 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.610+0000 7f3968ddb640 10 monclient: dump: 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: shutdown 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- 172.21.15.45:0/1377687584 >> v1:172.21.15.45:6790/0 conn(0x7f39641347d0 legacy=0x7f39641402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- 172.21.15.45:0/1377687584 shutdown_connections 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- 172.21.15.45:0/1377687584 >> 172.21.15.45:0/1377687584 conn(0x7f39640a2bd0 msgr2=0x7f39640a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- 172.21.15.45:0/1377687584 shutdown_connections 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- 172.21.15.45:0/1377687584 wait complete. 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: build_initial_monmap 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: monmap: 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 librados: starting msgr at 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 librados: starting objecter 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 Processor -- start 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- start start 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 librados: setting wanted keys 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 librados: calling monclient init 2024-09-16T03:20:31.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: init 2024-09-16T03:20:31.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: _reopen_session rank -1 2024-09-16T03:20:31.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient(hunting): picked mon.noname-c con 0x7f396409c570 addr v1:172.21.15.45:6790/0 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient(hunting): picked mon.noname-a con 0x7f3964134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient(hunting): picked mon.noname-b con 0x7f39641347d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3964133410 con 0x7f3964134320 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39641336f0 con 0x7f39641347d0 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3964130b80 con 0x7f396409c570 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient(hunting): _renew_subs 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient(hunting): authenticate will time out at 1922.266301s 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f39695dc640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f3964134320 legacy=0x7f3964130470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6789/0 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f3969ddd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f396409c570 0x7f396412fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46556/0 (socket says 172.21.15.45:46556) 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f3969ddd640 1 -- 172.21.15.45:0/2655070790 learned_addr learned my addr 172.21.15.45:0/2655070790 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 client.?.objecter ms_handle_connect 0x7f396409c570 2024-09-16T03:20:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 561360177 0 0) 0x7f3964130b80 con 0x7f396409c570 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient(hunting): my global_id is 144416 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3948004ec0 con 0x7f396409c570 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 client.?.objecter ms_handle_connect 0x7f39641347d0 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1612863151 0 0) 0x7f3948004ec0 con 0x7f396409c570 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3964130b80 con 0x7f396409c570 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3960001dc0 con 0x7f396409c570 2024-09-16T03:20:31.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient(hunting): dump: 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:20:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2009457876 0 0) 0x7f3964130b80 con 0x7f396409c570 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 >> v1:172.21.15.46:6789/0 conn(0x7f39641347d0 legacy=0x7f39641d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 >> v1:172.21.15.45:6789/0 conn(0x7f3964134320 legacy=0x7f3964130470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 monclient: found mon.c 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396409d2c0 con 0x7f396409c570 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 5 monclient: authenticate success, global_id 144416 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: _renew_subs 2024-09-16T03:20:31.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f396409bf70 con 0x7f396409c570 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 client.144416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: _renew_subs 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f39641321d0 con 0x7f396409c570 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 librados: init done 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3960002080 con 0x7f396409c570 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3960003df0 con 0x7f396409c570 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395a7fc640 10 monclient: dump: 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f395bfff640 4 set_mon_vals no callback set 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.614+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f394c0065d0 con 0x7f394c002870 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f3960004010 con 0x7f396409c570 2024-09-16T03:20:31.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 client.144416.objecter ms_handle_connect 0x7f394c002870 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 client.144416.objecter resend_mon_ops 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(632..632 src has 99..632) ==== 5925+0+0 (unknown 723040834 0 0) 0x7f396004bee0 con 0x7f396409c570 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 client.144416.objecter ms_dispatch 0x7f3964000ba0 osd_map(632..632 src has 99..632) 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 3 client.144416.objecter handle_osd_map got epochs [632,632] > 0 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 3 client.144416.objecter handle_osd_map decoding full epoch 632 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 20 client.144416.objecter dump_active .. 0 homeless 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3362048244 0 0) 0x7f394c0065d0 con 0x7f394c002870 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 monclient(hunting): my global_id is 164188 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3964130b80 con 0x7f394c002870 2024-09-16T03:20:31.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f396a5de640 10 monclient: get_auth_request con 0x7f394803deb0 auth_method 0 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1646057537 0 0) 0x7f3964130b80 con 0x7f394c002870 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f394c0065d0 con 0x7f394c002870 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3954003fb0 con 0x7f394c002870 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 monclient: dump: 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.634 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 client.144416.objecter ms_handle_connect 0x7f394803deb0 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 20107626 0 0) 0x7f394c0065d0 con 0x7f394c002870 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f394c00a170 con 0x7f394c002870 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f39540041f0 con 0x7f394c002870 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 >> v1:172.21.15.46:6789/0 conn(0x7f394c002870 legacy=0x7f394c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.618+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 >> v1:172.21.15.46:6789/0 conn(0x7f394c002870 legacy=0x7f394c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f394c009e60 con 0x7f394c009120 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 client.144416.objecter ms_handle_connect 0x7f394c009120 2024-09-16T03:20:31.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 client.144416.objecter resend_mon_ops 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286656533 0 0) 0x7f394c009e60 con 0x7f394c009120 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 monclient(hunting): my global_id is 164191 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f394c0065d0 con 0x7f394c009120 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2792524591 0 0) 0x7f394c0065d0 con 0x7f394c009120 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f394c009e60 con 0x7f394c009120 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f39540047f0 con 0x7f394c009120 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:20:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 monclient: dump: 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1430088750 0 0) 0x7f394c009e60 con 0x7f394c009120 2024-09-16T03:20:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f394c00c400 con 0x7f394c009120 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4125 (unknown 0 0 844250611) 0x7f3954006d50 con 0x7f394c009120 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 >> v1:172.21.15.46:6789/0 conn(0x7f394c009120 legacy=0x7f394c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f395a7fc640 1 -- 172.21.15.45:0/2655070790 >> v1:172.21.15.46:6789/0 conn(0x7f394c009120 legacy=0x7f394c0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 66, 2024-09-16T03:20:31.639 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 33939, 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:31.640 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:31.641 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:20:31.642 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:20:31.643 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.644 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.645 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:31.646 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:20:31.647 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:20:31.648 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:20:31.650 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:20:31.651 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:20:31.651 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:20:31.651 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:20:31.651 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:20:31.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:20:31.651 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:20:31.651 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 10 librados: watch_flush enter 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 10 librados: watch_flush exit 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 20 client.144416.objecter shutdown clearing up homeless session... 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 10 client.144416.objecter successfully canceled tick 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 >> v1:172.21.15.46:6812/149360280 conn(0x7f394803deb0 legacy=0x7f3948040370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 10 monclient: shutdown 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 >> v1:172.21.15.45:6790/0 conn(0x7f396409c570 legacy=0x7f396412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396a5de640 1 -- 172.21.15.45:0/2655070790 reap_dead start 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396a5de640 1 -- 172.21.15.45:0/2655070790 reap_dead start 2024-09-16T03:20:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 shutdown_connections 2024-09-16T03:20:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 >> 172.21.15.45:0/2655070790 conn(0x7f39640a2bd0 msgr2=0x7f396409f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:20:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 shutdown_connections 2024-09-16T03:20:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 1 -- 172.21.15.45:0/2655070790 wait complete. 2024-09-16T03:20:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:20:31.626+0000 7f396c041640 1 librados: shutdown 2024-09-16T03:20:31.654 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:20:31.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:22:01.654 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T03:22:01.655 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:22:01.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:22:01.660 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:22:01.660 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:22:01.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:22:01.762 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:01.760+0000 7f7281f87e00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 665 -> 0 (sync_force? mon rebuild?) 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient: get_monmap_and_config 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient: build_initial_monmap 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient: monmap: 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:01.789269+0000 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:01.789269+0000 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:01.788 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:01.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 1 Processor -- start 2024-09-16T03:22:01.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 1 -- start start 2024-09-16T03:22:01.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient: init 2024-09-16T03:22:01.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:01.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:22:01.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f501c0a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:01.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f501c0a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f501c073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501c09d9e0 con 0x7f501c0a4d20 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501c09dc20 con 0x7f501c073d40 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501c073360 con 0x7f501c0a5810 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient(hunting): authenticate will time out at 2012.438447s 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502103d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f501c0a5810 0x7f501c13cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50164/0 (socket says 172.21.15.45:50164) 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502103d640 1 -- 172.21.15.45:0/1530793676 learned_addr learned my addr 172.21.15.45:0/1530793676 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502183e640 1 --1- 172.21.15.45:0/1530793676 >> v1:172.21.15.45:6789/0 conn(0x7f501c0a4d20 0x7f501c0a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44530/0 (socket says 172.21.15.45:44530) 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 -- 172.21.15.45:0/1530793676 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2226840721 0 0) 0x7f501c073360 con 0x7f501c0a5810 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient(hunting): _negotiate 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient(hunting): my global_id is 144419 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 -- 172.21.15.45:0/1530793676 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5008003530 con 0x7f501c0a5810 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 -- 172.21.15.45:0/1530793676 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3843782074 0 0) 0x7f5008003530 con 0x7f501c0a5810 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 -- 172.21.15.45:0/1530793676 >> v1:172.21.15.46:6789/0 conn(0x7f501c073d40 legacy=0x7f501c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 -- 172.21.15.45:0/1530793676 >> v1:172.21.15.45:6789/0 conn(0x7f501c0a4d20 legacy=0x7f501c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: _finish_hunting 0 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 monclient: found mon.noname-b 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 -- 172.21.15.45:0/1530793676 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501c09cf30 con 0x7f501c0a5810 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: _finish_auth 0 2024-09-16T03:22:01.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: _check_auth_tickets 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 5 monclient: authenticate success, global_id 144419 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 -- 172.21.15.45:0/1530793676 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f500c0016a0 con 0x7f501c0a5810 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: dump: 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 -- 172.21.15.45:0/1530793676 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f500c0018c0 con 0x7f501c0a5810 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f5023aa2640 4 set_mon_vals no callback set 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 1 -- 172.21.15.45:0/1530793676 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f500c001d50 con 0x7f501c0a5810 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.784+0000 7f502083c640 10 monclient: dump: 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: shutdown 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- 172.21.15.45:0/1530793676 >> v1:172.21.15.45:6790/0 conn(0x7f501c0a5810 legacy=0x7f501c13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- 172.21.15.45:0/1530793676 shutdown_connections 2024-09-16T03:22:01.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- 172.21.15.45:0/1530793676 >> 172.21.15.45:0/1530793676 conn(0x7f501c0aa720 msgr2=0x7f501c0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- 172.21.15.45:0/1530793676 shutdown_connections 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- 172.21.15.45:0/1530793676 wait complete. 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: build_initial_monmap 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: monmap: 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:01.797 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:01.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 librados: starting msgr at 2024-09-16T03:22:01.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 librados: starting objecter 2024-09-16T03:22:01.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 Processor -- start 2024-09-16T03:22:01.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- start start 2024-09-16T03:22:01.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 librados: setting wanted keys 2024-09-16T03:22:01.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 librados: calling monclient init 2024-09-16T03:22:01.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: init 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f501c073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f501c0a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f501c0a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501c138aa0 con 0x7f501c0a4d20 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501c140190 con 0x7f501c073d40 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501c140470 con 0x7f501c0a5810 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient(hunting): authenticate will time out at 2012.442299s 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f502103d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f501c0a4d20 0x7f501c13d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44544/0 (socket says 172.21.15.45:44544) 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f502103d640 1 -- 172.21.15.45:0/980199078 learned_addr learned my addr 172.21.15.45:0/980199078 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 client.?.objecter ms_handle_connect 0x7f501c0a4d20 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 client.?.objecter ms_handle_connect 0x7f501c0a5810 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 606020134 0 0) 0x7f501c140470 con 0x7f501c0a5810 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): my global_id is 144422 2024-09-16T03:22:01.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5000004ef0 con 0x7f501c0a5810 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 client.?.objecter ms_handle_connect 0x7f501c073d40 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2776905984 0 0) 0x7f501c140190 con 0x7f501c073d40 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): my global_id is 164197 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f501c140470 con 0x7f501c073d40 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3809520370 0 0) 0x7f5000004ef0 con 0x7f501c0a5810 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f501c140190 con 0x7f501c0a5810 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5014001bc0 con 0x7f501c0a5810 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:01.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): dump: 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:01.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4286516595 0 0) 0x7f501c140190 con 0x7f501c0a5810 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 >> v1:172.21.15.46:6789/0 conn(0x7f501c073d40 legacy=0x7f501c13ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 >> v1:172.21.15.45:6789/0 conn(0x7f501c0a4d20 legacy=0x7f501c13d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient: _finish_hunting 0 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 monclient: found mon.c 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501c13d820 con 0x7f501c0a5810 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient: _finish_auth 0 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient: _check_auth_tickets 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 5 monclient: authenticate success, global_id 144422 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: _renew_subs 2024-09-16T03:22:01.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f501c13f8e0 con 0x7f501c0a5810 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 client.144422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: _renew_subs 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f501c13ef50 con 0x7f501c0a5810 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5023aa2640 1 librados: init done 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50140039a0 con 0x7f501c0a5810 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5014003e30 con 0x7f501c0a5810 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f5013fff640 4 set_mon_vals no callback set 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.788+0000 7f50127fc640 10 monclient: dump: 2024-09-16T03:22:01.807 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f5023aa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f5023aa2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f5023aa2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:01.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5004005090 con 0x7f501c0a5810 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f5014004050 con 0x7f501c0a5810 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(693..693 src has 99..693) ==== 5917+0+0 (unknown 3513558623 0 0) 0x7f501404aa80 con 0x7f501c0a5810 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f50127fc640 10 client.144422.objecter ms_dispatch 0x7f501c000ba0 osd_map(693..693 src has 99..693) 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f50127fc640 3 client.144422.objecter handle_osd_map got epochs [693,693] > 0 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f50127fc640 3 client.144422.objecter handle_osd_map decoding full epoch 693 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f50127fc640 20 client.144422.objecter dump_active .. 0 homeless 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f502183e640 10 monclient: get_auth_request con 0x7f500003e170 auth_method 0 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.792+0000 7f50127fc640 10 client.144422.objecter ms_handle_connect 0x7f500003e170 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.800+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5014015bd0 con 0x7f501c0a5810 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.800+0000 7f50127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:01.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:01.800+0000 7f50127fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:02.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:22:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5004005380 con 0x7f501c0a5810 2024-09-16T03:22:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f50127fc640 1 -- 172.21.15.45:0/980199078 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2361251403) 0x7f50140154d0 con 0x7f501c0a5810 2024-09-16T03:22:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f50127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f50127fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:02.072 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:22:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 10 librados: watch_flush enter 2024-09-16T03:22:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 10 librados: watch_flush exit 2024-09-16T03:22:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 20 client.144422.objecter shutdown clearing up homeless session... 2024-09-16T03:22:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 10 client.144422.objecter successfully canceled tick 2024-09-16T03:22:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 >> v1:172.21.15.46:6812/149360280 conn(0x7f500003e170 legacy=0x7f5000040630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 10 monclient: shutdown 2024-09-16T03:22:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 >> v1:172.21.15.45:6790/0 conn(0x7f501c0a5810 legacy=0x7f501c1d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 shutdown_connections 2024-09-16T03:22:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 >> 172.21.15.45:0/980199078 conn(0x7f501c0aa720 msgr2=0x7f501c0a3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 shutdown_connections 2024-09-16T03:22:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 1 -- 172.21.15.45:0/980199078 wait complete. 2024-09-16T03:22:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.068+0000 7f5023aa2640 1 librados: shutdown 2024-09-16T03:22:02.088 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:22:02.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:22:02.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:22:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient: get_monmap_and_config 2024-09-16T03:22:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient: build_initial_monmap 2024-09-16T03:22:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient: monmap: 2024-09-16T03:22:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:02.216400+0000 2024-09-16T03:22:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:02.216400+0000 2024-09-16T03:22:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:02.215 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 Processor -- start 2024-09-16T03:22:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 -- start start 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient: init 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient(hunting): picked mon.noname-a con 0x7f8998137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient(hunting): picked mon.noname-c con 0x7f8998138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient(hunting): picked mon.noname-b con 0x7f899813e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89980a4f90 con 0x7f8998137d20 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89980a4cb0 con 0x7f8998138100 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f899809dc70 con 0x7f899813e2a0 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient(hunting): authenticate will time out at 2012.865762s 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89977fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8998137d20 0x7f89980a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44552/0 (socket says 172.21.15.45:44552) 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89977fe640 1 -- 172.21.15.45:0/701568388 learned_addr learned my addr 172.21.15.45:0/701568388 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1583177908 0 0) 0x7f899809dc70 con 0x7f899813e2a0 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient(hunting): my global_id is 144425 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8980003530 con 0x7f899813e2a0 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776423184 0 0) 0x7f89980a4f90 con 0x7f8998137d20 2024-09-16T03:22:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient(hunting): my global_id is 174099 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f899809dc70 con 0x7f8998137d20 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3884912653 0 0) 0x7f8980003530 con 0x7f899813e2a0 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 >> v1:172.21.15.46:6789/0 conn(0x7f8998138100 legacy=0x7f899813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 >> v1:172.21.15.45:6789/0 conn(0x7f8998137d20 legacy=0x7f89980a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: _finish_hunting 0 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 monclient: found mon.noname-b 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f899809cc50 con 0x7f899813e2a0 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: _finish_auth 0 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: _check_auth_tickets 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f89880016a0 con 0x7f899813e2a0 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: dump: 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 5 monclient: authenticate success, global_id 144425 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89880018c0 con 0x7f899813e2a0 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 1 -- 172.21.15.45:0/701568388 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8988001d50 con 0x7f899813e2a0 2024-09-16T03:22:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 4 set_mon_vals no callback set 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f89967fc640 10 monclient: dump: 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.229 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 10 monclient: shutdown 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 -- 172.21.15.45:0/701568388 >> v1:172.21.15.45:6790/0 conn(0x7f899813e2a0 legacy=0x7f8998140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 -- 172.21.15.45:0/701568388 shutdown_connections 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 -- 172.21.15.45:0/701568388 >> 172.21.15.45:0/701568388 conn(0x7f89980aa110 msgr2=0x7f89980aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 -- 172.21.15.45:0/701568388 shutdown_connections 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.212+0000 7f899db2e640 1 -- 172.21.15.45:0/701568388 wait complete. 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient: build_initial_monmap 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient: monmap: 2024-09-16T03:22:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:02.231 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:02.231 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:02.231 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:02.231 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:02.231 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.231 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:02.231 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:02.231 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 librados: starting msgr at 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 librados: starting objecter 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 Processor -- start 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 -- start start 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 librados: setting wanted keys 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 librados: calling monclient init 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient: init 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient(hunting): picked mon.noname-a con 0x7f8998137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient(hunting): picked mon.noname-c con 0x7f8998138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient(hunting): picked mon.noname-b con 0x7f899813e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89980a2120 con 0x7f8998137d20 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8998141af0 con 0x7f899813e2a0 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8998138a50 con 0x7f8998138100 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient(hunting): authenticate will time out at 2012.868945s 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f8996ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8998138100 0x7f89980a1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50198/0 (socket says 172.21.15.45:50198) 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f8996ffd640 1 -- 172.21.15.45:0/72415083 learned_addr learned my addr 172.21.15.45:0/72415083 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 client.?.objecter ms_handle_connect 0x7f8998137d20 2024-09-16T03:22:02.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 client.?.objecter ms_handle_connect 0x7f8998138100 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 616569565 0 0) 0x7f89980a2120 con 0x7f8998137d20 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): my global_id is 174102 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8984004ec0 con 0x7f8998137d20 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2295451921 0 0) 0x7f8998138a50 con 0x7f8998138100 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): my global_id is 144428 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89980a2120 con 0x7f8998138100 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 client.?.objecter ms_handle_connect 0x7f899813e2a0 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1913828459 0 0) 0x7f8984004ec0 con 0x7f8998137d20 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8998138a50 con 0x7f8998137d20 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8990001660 con 0x7f8998137d20 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): dump: 2024-09-16T03:22:02.235 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3806923645 0 0) 0x7f89980a2120 con 0x7f8998138100 2024-09-16T03:22:02.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8984004ec0 con 0x7f8998138100 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f898c001980 con 0x7f8998138100 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): dump: 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.237 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265478221 0 0) 0x7f8998141af0 con 0x7f899813e2a0 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): my global_id is 164203 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89980a2120 con 0x7f899813e2a0 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3853275725 0 0) 0x7f8998138a50 con 0x7f8998137d20 2024-09-16T03:22:02.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 >> v1:172.21.15.45:6790/0 conn(0x7f8998138100 legacy=0x7f89980a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 >> v1:172.21.15.46:6789/0 conn(0x7f899813e2a0 legacy=0x7f89981d4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 monclient: found mon.a 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89980a32f0 con 0x7f8998137d20 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: _finish_auth 0 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 5 monclient: authenticate success, global_id 174102 2024-09-16T03:22:02.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient: _renew_subs 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8998137750 con 0x7f8998137d20 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 client.174102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient: _renew_subs 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8998140e80 con 0x7f8998137d20 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 librados: init done 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8990004b70 con 0x7f8998137d20 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:02.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8990004ec0 con 0x7f8998137d20 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f8995ffb640 4 set_mon_vals no callback set 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: dump: 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.241 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f89780065d0 con 0x7f8978002870 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f8990010f30 con 0x7f8998137d20 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 client.174102.objecter ms_handle_connect 0x7f8978002870 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 client.174102.objecter resend_mon_ops 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(694..694 src has 99..694) ==== 5930+0+0 (unknown 2599890563 0 0) 0x7f899004bc50 con 0x7f8998137d20 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 client.174102.objecter ms_dispatch 0x7f8998000ba0 osd_map(694..694 src has 99..694) 2024-09-16T03:22:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 3 client.174102.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 3 client.174102.objecter handle_osd_map decoding full epoch 694 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 20 client.174102.objecter dump_active .. 0 homeless 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 305591243 0 0) 0x7f89780065d0 con 0x7f8978002870 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient(hunting): my global_id is 174105 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8998138a50 con 0x7f8978002870 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3532356853 0 0) 0x7f8998138a50 con 0x7f8978002870 2024-09-16T03:22:02.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89780065d0 con 0x7f8978002870 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f898c0037c0 con 0x7f8978002870 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 10 monclient: dump: 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f8997fff640 10 monclient: get_auth_request con 0x7f898403e2b0 auth_method 0 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1671429796 0 0) 0x7f89780065d0 con 0x7f8978002870 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.216+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f897800a170 con 0x7f8978002870 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.220+0000 7f897ffff640 10 client.174102.objecter ms_handle_connect 0x7f898403e2b0 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.220+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f898c0057d0 con 0x7f8978002870 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.220+0000 7f897ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.220+0000 7f897ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:02.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.220+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 >> v1:172.21.15.45:6789/0 conn(0x7f8978002870 legacy=0x7f8978004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.220+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 >> v1:172.21.15.45:6789/0 conn(0x7f8978002870 legacy=0x7f8978004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:02.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:22:02.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8978009e60 con 0x7f8978009120 2024-09-16T03:22:02.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 client.174102.objecter ms_handle_connect 0x7f8978009120 2024-09-16T03:22:02.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 client.174102.objecter resend_mon_ops 2024-09-16T03:22:02.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388525255 0 0) 0x7f8978009e60 con 0x7f8978009120 2024-09-16T03:22:02.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 monclient(hunting): my global_id is 174108 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89780065d0 con 0x7f8978009120 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 151638225 0 0) 0x7f89780065d0 con 0x7f8978009120 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8978009e60 con 0x7f8978009120 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f898c003da0 con 0x7f8978009120 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 monclient: dump: 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.248 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2417197407 0 0) 0x7f8978009e60 con 0x7f8978009120 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f897800c3b0 con 0x7f8978009120 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3967 (unknown 0 0 2858130362) 0x7f898c003de0 con 0x7f8978009120 2024-09-16T03:22:02.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:22:02.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:02.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 >> v1:172.21.15.45:6789/0 conn(0x7f8978009120 legacy=0x7f89780095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f897ffff640 1 -- 172.21.15.45:0/72415083 >> v1:172.21.15.45:6789/0 conn(0x7f8978009120 legacy=0x7f89780095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:02.250 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:22:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:22:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 70, 2024-09-16T03:22:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 483, 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:02.253 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:22:02.254 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:02.255 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:22:02.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.257 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:02.258 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:22:02.260 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.261 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:02.262 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:02.263 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:22:02.263 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.263 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.263 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:22:02.263 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 10 librados: watch_flush enter 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 10 librados: watch_flush exit 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 20 client.174102.objecter shutdown clearing up homeless session... 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 10 client.174102.objecter successfully canceled tick 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 >> v1:172.21.15.46:6812/149360280 conn(0x7f898403e2b0 legacy=0x7f8984040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 10 monclient: shutdown 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 >> v1:172.21.15.45:6789/0 conn(0x7f8998137d20 legacy=0x7f89980a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f8997fff640 1 -- 172.21.15.45:0/72415083 reap_dead start 2024-09-16T03:22:02.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f8997fff640 1 -- 172.21.15.45:0/72415083 reap_dead start 2024-09-16T03:22:02.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 shutdown_connections 2024-09-16T03:22:02.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 >> 172.21.15.45:0/72415083 conn(0x7f89980aa110 msgr2=0x7f89980731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:02.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 shutdown_connections 2024-09-16T03:22:02.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 1 -- 172.21.15.45:0/72415083 wait complete. 2024-09-16T03:22:02.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.224+0000 7f899db2e640 1 librados: shutdown 2024-09-16T03:22:02.266 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:22:02.266 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:22:02.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.376+0000 7f6a5f3e0640 10 monclient: get_monmap_and_config 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.376+0000 7f6a5f3e0640 10 monclient: build_initial_monmap 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.376+0000 7f6a5f3e0640 10 monclient: monmap: 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:02.384166+0000 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:02.384166+0000 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.382 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:02.383 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:02.383 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 Processor -- start 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 -- start start 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient: init 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient(hunting): picked mon.noname-a con 0x7f6a580a7f80 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient(hunting): picked mon.noname-b con 0x7f6a580a8360 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient(hunting): picked mon.noname-c con 0x7f6a5813de80 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a580a4d60 con 0x7f6a580a7f80 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a580a5040 con 0x7f6a5813de80 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a580a5320 con 0x7f6a580a8360 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient(hunting): authenticate will time out at 2013.033901s 2024-09-16T03:22:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5c97b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6a580a8360 0x7f6a580a8790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50212/0 (socket says 172.21.15.45:50212) 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5d17c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6a580a7f80 0x7f6a580a5d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44592/0 (socket says 172.21.15.45:44592) 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5c97b640 1 -- 172.21.15.45:0/263239567 learned_addr learned my addr 172.21.15.45:0/263239567 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050271699 0 0) 0x7f6a580a5320 con 0x7f6a580a8360 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient(hunting): my global_id is 144431 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a3c003530 con 0x7f6a580a8360 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4037675321 0 0) 0x7f6a580a4d60 con 0x7f6a580a7f80 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient(hunting): my global_id is 174111 2024-09-16T03:22:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a580a5320 con 0x7f6a580a7f80 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 247379508 0 0) 0x7f6a3c003530 con 0x7f6a580a8360 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 >> v1:172.21.15.46:6789/0 conn(0x7f6a5813de80 legacy=0x7f6a58140270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 >> v1:172.21.15.45:6789/0 conn(0x7f6a580a7f80 legacy=0x7f6a580a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: _finish_hunting 0 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 monclient: found mon.noname-b 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a580a8ea0 con 0x7f6a580a8360 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: _finish_auth 0 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: _check_auth_tickets 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a440015d0 con 0x7f6a580a8360 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: dump: 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 5 monclient: authenticate success, global_id 144431 2024-09-16T03:22:02.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a44001770 con 0x7f6a580a8360 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 1 -- 172.21.15.45:0/263239567 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a44001b80 con 0x7f6a580a8360 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a57fff640 10 monclient: dump: 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.390 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 4 set_mon_vals no callback set 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 10 monclient: shutdown 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/263239567 >> v1:172.21.15.45:6790/0 conn(0x7f6a580a8360 legacy=0x7f6a580a8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/263239567 shutdown_connections 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/263239567 >> 172.21.15.45:0/263239567 conn(0x7f6a580aa180 msgr2=0x7f6a580aa590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/263239567 shutdown_connections 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.380+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/263239567 wait complete. 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient: build_initial_monmap 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient: monmap: 2024-09-16T03:22:02.391 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:02.392 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 librados: starting msgr at 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 librados: starting objecter 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 Processor -- start 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 -- start start 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 librados: setting wanted keys 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 librados: calling monclient init 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient: init 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient(hunting): picked mon.noname-c con 0x7f6a580a7f80 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient(hunting): picked mon.noname-a con 0x7f6a580a8360 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient(hunting): picked mon.noname-b con 0x7f6a5813de80 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a5813cc00 con 0x7f6a580a8360 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a58074c20 con 0x7f6a5813de80 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a581413b0 con 0x7f6a580a7f80 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient(hunting): authenticate will time out at 2013.037130s 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5c97b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6a580a8360 0x7f6a581d0df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44604/0 (socket says 172.21.15.45:44604) 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5d17c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6a580a7f80 0x7f6a5813d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50224/0 (socket says 172.21.15.45:50224) 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5c97b640 1 -- 172.21.15.45:0/604436804 learned_addr learned my addr 172.21.15.45:0/604436804 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 client.?.objecter ms_handle_connect 0x7f6a580a8360 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 client.?.objecter ms_handle_connect 0x7f6a580a7f80 2024-09-16T03:22:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3771934949 0 0) 0x7f6a581413b0 con 0x7f6a580a7f80 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): my global_id is 144434 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a40004ec0 con 0x7f6a580a7f80 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3512677733 0 0) 0x7f6a5813cc00 con 0x7f6a580a8360 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): my global_id is 174114 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a581413b0 con 0x7f6a580a8360 2024-09-16T03:22:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 client.?.objecter ms_handle_connect 0x7f6a5813de80 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1653037112 0 0) 0x7f6a40004ec0 con 0x7f6a580a7f80 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a5813cc00 con 0x7f6a580a7f80 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 235980986 0 0) 0x7f6a581413b0 con 0x7f6a580a8360 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a40004ec0 con 0x7f6a580a8360 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a50001660 con 0x7f6a580a7f80 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): dump: 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a44001b00 con 0x7f6a580a8360 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): dump: 2024-09-16T03:22:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2756369878 0 0) 0x7f6a5813cc00 con 0x7f6a580a7f80 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 >> v1:172.21.15.46:6789/0 conn(0x7f6a5813de80 legacy=0x7f6a581d4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 >> v1:172.21.15.45:6789/0 conn(0x7f6a580a8360 legacy=0x7f6a581d0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:22:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 monclient: found mon.c 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a580a1370 con 0x7f6a580a7f80 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: _finish_auth 0 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 5 monclient: authenticate success, global_id 144434 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient: _renew_subs 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a5813c920 con 0x7f6a580a7f80 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 client.144434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient: _renew_subs 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a581d5e10 con 0x7f6a580a7f80 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 librados: init done 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a50004b70 con 0x7f6a580a7f80 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a50004ec0 con 0x7f6a580a7f80 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a577fe640 4 set_mon_vals no callback set 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: dump: 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6a340065d0 con 0x7f6a34002870 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 client.144434.objecter ms_handle_connect 0x7f6a34002870 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 client.144434.objecter resend_mon_ops 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f6a50010f30 con 0x7f6a580a7f80 2024-09-16T03:22:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137944227 0 0) 0x7f6a340065d0 con 0x7f6a34002870 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient(hunting): my global_id is 144437 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a5813cc00 con 0x7f6a34002870 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(694..694 src has 99..694) ==== 5930+0+0 (unknown 2599890563 0 0) 0x7f6a5004bb60 con 0x7f6a580a7f80 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 client.144434.objecter ms_dispatch 0x7f6a58000ba0 osd_map(694..694 src has 99..694) 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 3 client.144434.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 3 client.144434.objecter handle_osd_map decoding full epoch 694 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 20 client.144434.objecter dump_active .. 0 homeless 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3867197020 0 0) 0x7f6a5813cc00 con 0x7f6a34002870 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a340065d0 con 0x7f6a34002870 2024-09-16T03:22:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a44004cd0 con 0x7f6a34002870 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 10 monclient: dump: 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.408 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a5d97d640 10 monclient: get_auth_request con 0x7f6a4003df40 auth_method 0 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2785948069 0 0) 0x7f6a340065d0 con 0x7f6a34002870 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.384+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a3400a170 con 0x7f6a34002870 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.388+0000 7f6a55ffb640 10 client.144434.objecter ms_handle_connect 0x7f6a4003df40 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.388+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6a44007480 con 0x7f6a34002870 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.388+0000 7f6a55ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.388+0000 7f6a55ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.388+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 >> v1:172.21.15.45:6790/0 conn(0x7f6a34002870 legacy=0x7f6a34004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.388+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 >> v1:172.21.15.45:6790/0 conn(0x7f6a34002870 legacy=0x7f6a34004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6a34009e60 con 0x7f6a34009120 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 client.144434.objecter ms_handle_connect 0x7f6a34009120 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 client.144434.objecter resend_mon_ops 2024-09-16T03:22:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 176601668 0 0) 0x7f6a34009e60 con 0x7f6a34009120 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 monclient(hunting): my global_id is 144440 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a340065d0 con 0x7f6a34009120 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3356750270 0 0) 0x7f6a340065d0 con 0x7f6a34009120 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a34009e60 con 0x7f6a34009120 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6a44005560 con 0x7f6a34009120 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 monclient: dump: 2024-09-16T03:22:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 569450844 0 0) 0x7f6a34009e60 con 0x7f6a34009120 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6a3400c400 con 0x7f6a34009120 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 1200824554) 0x7f6a440076b0 con 0x7f6a34009120 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:22:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 >> v1:172.21.15.45:6790/0 conn(0x7f6a34009120 legacy=0x7f6a340095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a55ffb640 1 -- 172.21.15.45:0/604436804 >> v1:172.21.15.45:6790/0 conn(0x7f6a34009120 legacy=0x7f6a340095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:22:02.412 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 70, 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 347466, 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:22:02.413 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.414 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:22:02.416 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:02.417 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:02.418 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:02.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.420 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:22:02.421 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:22:02.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:22:02.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.424 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:02.424 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:02.424 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:22:02.424 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.424 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.424 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:22:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:22:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 10 librados: watch_flush enter 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 10 librados: watch_flush exit 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 20 client.144434.objecter shutdown clearing up homeless session... 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 10 client.144434.objecter successfully canceled tick 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 >> v1:172.21.15.46:6812/149360280 conn(0x7f6a4003df40 legacy=0x7f6a40040400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 10 monclient: shutdown 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 >> v1:172.21.15.45:6790/0 conn(0x7f6a580a7f80 legacy=0x7f6a5813d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5d97d640 1 -- 172.21.15.45:0/604436804 reap_dead start 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5d97d640 1 -- 172.21.15.45:0/604436804 reap_dead start 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 shutdown_connections 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 >> 172.21.15.45:0/604436804 conn(0x7f6a580aa180 msgr2=0x7f6a580a0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 shutdown_connections 2024-09-16T03:22:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 1 -- 172.21.15.45:0/604436804 wait complete. 2024-09-16T03:22:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.392+0000 7f6a5f3e0640 1 librados: shutdown 2024-09-16T03:22:02.427 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:22:02.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:22:02.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:22:02.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient: get_monmap_and_config 2024-09-16T03:22:02.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient: build_initial_monmap 2024-09-16T03:22:02.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient: monmap: 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:02.541588+0000 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:02.541588+0000 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:02.540 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 1 Processor -- start 2024-09-16T03:22:02.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 1 -- start start 2024-09-16T03:22:02.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient: init 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient(hunting): picked mon.noname-b con 0x7fe8e4137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient(hunting): picked mon.noname-a con 0x7fe8e4138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient(hunting): picked mon.noname-c con 0x7fe8e413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8e40a4f90 con 0x7fe8e4138100 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8e40a4cb0 con 0x7fe8e413e2a0 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8e409dc70 con 0x7fe8e4137d20 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8ea719640 10 monclient(hunting): authenticate will time out at 2013.191845s 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8e3fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe8e4137d20 0x7fe8e40a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50252/0 (socket says 172.21.15.45:50252) 2024-09-16T03:22:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8e3fff640 1 -- 172.21.15.45:0/2729053267 learned_addr learned my addr 172.21.15.45:0/2729053267 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8dbfff640 1 -- 172.21.15.45:0/2729053267 >> v1:172.21.15.45:6789/0 conn(0x7fe8e4138100 legacy=0x7fe8e413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8dbfff640 1 -- 172.21.15.45:0/2729053267 >> v1:172.21.15.45:6789/0 conn(0x7fe8e4138100 legacy=0x7fe8e413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.536+0000 7fe8dbfff640 1 --1- 172.21.15.45:0/2729053267 >> v1:172.21.15.45:6789/0 conn(0x7fe8e4138100 0x7fe8e413cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2729053267 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2415023128 0 0) 0x7fe8e409dc70 con 0x7fe8e4137d20 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient(hunting): my global_id is 144443 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2729053267 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8d0003530 con 0x7fe8e4137d20 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2729053267 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3878570865 0 0) 0x7fe8d0003530 con 0x7fe8e4137d20 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2729053267 >> v1:172.21.15.46:6789/0 conn(0x7fe8e413e2a0 legacy=0x7fe8e4140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2729053267 >> v1:172.21.15.45:6789/0 conn(0x7fe8e4138100 legacy=0x7fe8e413cb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 monclient: found mon.noname-b 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2729053267 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e409cc50 con 0x7fe8e4137d20 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: _finish_auth 0 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2729053267 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe8d40016a0 con 0x7fe8e4137d20 2024-09-16T03:22:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: dump: 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 5 monclient: authenticate success, global_id 144443 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2729053267 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe8d4001970 con 0x7fe8e4137d20 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2729053267 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe8d40048b0 con 0x7fe8e4137d20 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 4 set_mon_vals no callback set 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e37fe640 10 monclient: dump: 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient: shutdown 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 -- 172.21.15.45:0/2729053267 >> v1:172.21.15.45:6790/0 conn(0x7fe8e4137d20 legacy=0x7fe8e40a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 -- 172.21.15.45:0/2729053267 shutdown_connections 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 -- 172.21.15.45:0/2729053267 >> 172.21.15.45:0/2729053267 conn(0x7fe8e40aa110 msgr2=0x7fe8e40aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:02.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 -- 172.21.15.45:0/2729053267 shutdown_connections 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 -- 172.21.15.45:0/2729053267 wait complete. 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient: build_initial_monmap 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient: monmap: 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.555 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 librados: starting msgr at 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 librados: starting objecter 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 Processor -- start 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 -- start start 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 librados: setting wanted keys 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 librados: calling monclient init 2024-09-16T03:22:02.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient: init 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient(hunting): picked mon.noname-c con 0x7fe8e4137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient(hunting): picked mon.noname-b con 0x7fe8e4138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient(hunting): picked mon.noname-a con 0x7fe8e413e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8e4141630 con 0x7fe8e413e2a0 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8e4140d90 con 0x7fe8e4138100 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8e413fe30 con 0x7fe8e4137d20 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8ea719640 10 monclient(hunting): authenticate will time out at 2013.195049s 2024-09-16T03:22:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e3fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe8e4137d20 0x7fe8e413e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50264/0 (socket says 172.21.15.45:50264) 2024-09-16T03:22:02.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e3fff640 1 -- 172.21.15.45:0/173676408 learned_addr learned my addr 172.21.15.45:0/173676408 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:02.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 client.?.objecter ms_handle_connect 0x7fe8e4137d20 2024-09-16T03:22:02.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 client.?.objecter ms_handle_connect 0x7fe8e413e2a0 2024-09-16T03:22:02.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781165510 0 0) 0x7fe8e413fe30 con 0x7fe8e4137d20 2024-09-16T03:22:02.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): my global_id is 144446 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8c4004ec0 con 0x7fe8e4137d20 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 875202918 0 0) 0x7fe8e4141630 con 0x7fe8e413e2a0 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): my global_id is 174117 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8e413fe30 con 0x7fe8e413e2a0 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 client.?.objecter ms_handle_connect 0x7fe8e4138100 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1336090500 0 0) 0x7fe8c4004ec0 con 0x7fe8e4137d20 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8e4141630 con 0x7fe8e4137d20 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe8d40014c0 con 0x7fe8e4137d20 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): dump: 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3333075383 0 0) 0x7fe8e413fe30 con 0x7fe8e413e2a0 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8c4004ec0 con 0x7fe8e413e2a0 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe8dc001c40 con 0x7fe8e413e2a0 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:02.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): dump: 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 123070937 0 0) 0x7fe8e4140d90 con 0x7fe8e4138100 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 10 monclient(hunting): my global_id is 164215 2024-09-16T03:22:02.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8e413fe30 con 0x7fe8e4138100 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.540+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 810604713 0 0) 0x7fe8e4141630 con 0x7fe8e4137d20 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 >> v1:172.21.15.46:6789/0 conn(0x7fe8e4138100 legacy=0x7fe8e413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 >> v1:172.21.15.45:6789/0 conn(0x7fe8e413e2a0 legacy=0x7fe8e41d4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: _finish_hunting 0 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 monclient: found mon.c 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e40a4330 con 0x7fe8e4137d20 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: _finish_auth 0 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:02.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 5 monclient: authenticate success, global_id 144446 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 10 monclient: _renew_subs 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e40a3cf0 con 0x7fe8e4137d20 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 10 client.144446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 10 monclient: _renew_subs 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe8e413f670 con 0x7fe8e4137d20 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 1 librados: init done 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe8d40011c0 con 0x7fe8e4137d20 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe8d40018c0 con 0x7fe8e4137d20 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e2ffd640 4 set_mon_vals no callback set 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: dump: 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe8c80065d0 con 0x7fe8c8002870 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fe8d4010cf0 con 0x7fe8e4137d20 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(694..694 src has 99..694) ==== 5930+0+0 (unknown 2599890563 0 0) 0x7fe8d404ba40 con 0x7fe8e4137d20 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 client.144446.objecter ms_dispatch 0x7fe8e4000ba0 osd_map(694..694 src has 99..694) 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 3 client.144446.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 3 client.144446.objecter handle_osd_map decoding full epoch 694 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 20 client.144446.objecter dump_active .. 0 homeless 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 client.144446.objecter ms_handle_connect 0x7fe8c8002870 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 client.144446.objecter resend_mon_ops 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e8cb6640 10 monclient: get_auth_request con 0x7fe8c403e1a0 auth_method 0 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2005134092 0 0) 0x7fe8c80065d0 con 0x7fe8c8002870 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient(hunting): my global_id is 164218 2024-09-16T03:22:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8e4141630 con 0x7fe8c8002870 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 client.144446.objecter ms_handle_connect 0x7fe8c403e1a0 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1716370067 0 0) 0x7fe8e4141630 con 0x7fe8c8002870 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8c80065d0 con 0x7fe8c8002870 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe8cc001800 con 0x7fe8c8002870 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: dump: 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.569 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.569 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.569 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.569 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2951117646 0 0) 0x7fe8c80065d0 con 0x7fe8c8002870 2024-09-16T03:22:02.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8c800a170 con 0x7fe8c8002870 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe8cc0015a0 con 0x7fe8c8002870 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 >> v1:172.21.15.46:6789/0 conn(0x7fe8c8002870 legacy=0x7fe8c8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.544+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 >> v1:172.21.15.46:6789/0 conn(0x7fe8c8002870 legacy=0x7fe8c8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8ea719640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe8c8009e60 con 0x7fe8c8009120 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 10 client.144446.objecter ms_handle_connect 0x7fe8c8009120 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 10 client.144446.objecter resend_mon_ops 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1086073072 0 0) 0x7fe8c8009e60 con 0x7fe8c8009120 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 10 monclient(hunting): my global_id is 164221 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8c80065d0 con 0x7fe8c8009120 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3913683768 0 0) 0x7fe8c80065d0 con 0x7fe8c8009120 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8c8009e60 con 0x7fe8c8009120 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe8cc004940 con 0x7fe8c8009120 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 10 monclient: dump: 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.572 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1797468864 0 0) 0x7fe8c8009e60 con 0x7fe8c8009120 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.552+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe8c800c400 con 0x7fe8c8009120 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 1330149731) 0x7fe8cc005e30 con 0x7fe8c8009120 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8e17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 >> v1:172.21.15.46:6789/0 conn(0x7fe8c8009120 legacy=0x7fe8c80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8e17fa640 1 -- 172.21.15.45:0/173676408 >> v1:172.21.15.46:6789/0 conn(0x7fe8c8009120 legacy=0x7fe8c80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 70, 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 124866, 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:22:02.574 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:02.575 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:02.576 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:22:02.577 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:02.578 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:02.579 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:02.580 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:22:02.582 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.583 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:02.584 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:22:02.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:02.585 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:02.585 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:02.585 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:02.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:02.585 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:02.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:02.585 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:22:02.585 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:22:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 10 librados: watch_flush enter 2024-09-16T03:22:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 10 librados: watch_flush exit 2024-09-16T03:22:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 20 client.144446.objecter shutdown clearing up homeless session... 2024-09-16T03:22:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 10 client.144446.objecter successfully canceled tick 2024-09-16T03:22:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 >> v1:172.21.15.46:6812/149360280 conn(0x7fe8c403e1a0 legacy=0x7fe8c4040660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 10 monclient: shutdown 2024-09-16T03:22:02.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 >> v1:172.21.15.45:6790/0 conn(0x7fe8e4137d20 legacy=0x7fe8e413e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8e8cb6640 1 -- 172.21.15.45:0/173676408 reap_dead start 2024-09-16T03:22:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8e8cb6640 1 -- 172.21.15.45:0/173676408 reap_dead start 2024-09-16T03:22:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 shutdown_connections 2024-09-16T03:22:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 >> 172.21.15.45:0/173676408 conn(0x7fe8e40aa110 msgr2=0x7fe8e40731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 shutdown_connections 2024-09-16T03:22:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 1 -- 172.21.15.45:0/173676408 wait complete. 2024-09-16T03:22:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.556+0000 7fe8ea719640 1 librados: shutdown 2024-09-16T03:22:02.588 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:22:02.588 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:22:02.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:22:02.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.704+0000 7f411edd8640 10 monclient: get_monmap_and_config 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.704+0000 7f411edd8640 10 monclient: build_initial_monmap 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.704+0000 7f411edd8640 10 monclient: monmap: 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:02.712005+0000 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:02.712005+0000 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:02.710 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:02.711 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 Processor -- start 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 -- start start 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient: init 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient(hunting): picked mon.noname-b con 0x7f4118137b40 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient(hunting): picked mon.noname-c con 0x7f411809e850 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient(hunting): picked mon.noname-a con 0x7f411813e170 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4118138630 con 0x7f411813e170 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f411806e2e0 con 0x7f411809e850 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f411806e8a0 con 0x7f4118137b40 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient(hunting): authenticate will time out at 2013.361596s 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411d375640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f411813e170 0x7f4118140560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44632/0 (socket says 172.21.15.45:44632) 2024-09-16T03:22:02.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411d375640 1 -- 172.21.15.45:0/3327501514 learned_addr learned my addr 172.21.15.45:0/3327501514 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:02.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1291526777 0 0) 0x7f4118138630 con 0x7f411813e170 2024-09-16T03:22:02.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): my global_id is 174120 2024-09-16T03:22:02.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4100003530 con 0x7f411813e170 2024-09-16T03:22:02.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4000290931 0 0) 0x7f411806e8a0 con 0x7f4118137b40 2024-09-16T03:22:02.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): my global_id is 144449 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4118138630 con 0x7f4118137b40 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 504645599 0 0) 0x7f411806e2e0 con 0x7f411809e850 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient(hunting): my global_id is 164224 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f411806e8a0 con 0x7f411809e850 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4143307993 0 0) 0x7f4118138630 con 0x7f4118137b40 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 >> v1:172.21.15.46:6789/0 conn(0x7f411809e850 legacy=0x7f411813ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 >> v1:172.21.15.45:6789/0 conn(0x7f411813e170 legacy=0x7f4118140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: _finish_hunting 0 2024-09-16T03:22:02.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 monclient: found mon.noname-b 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411806cd50 con 0x7f4118137b40 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: _finish_auth 0 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: _check_auth_tickets 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4108001a00 con 0x7f4118137b40 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 5 monclient: authenticate success, global_id 144449 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: dump: 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4108001ba0 con 0x7f4118137b40 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 1 -- 172.21.15.45:0/3327501514 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4108001fb0 con 0x7f4118137b40 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 4 set_mon_vals no callback set 2024-09-16T03:22:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f41177fe640 10 monclient: dump: 2024-09-16T03:22:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.720 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.720 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.720 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.720 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.720 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.720 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.720 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 10 monclient: shutdown 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 -- 172.21.15.45:0/3327501514 >> v1:172.21.15.45:6790/0 conn(0x7f4118137b40 legacy=0x7f4118137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 -- 172.21.15.45:0/3327501514 shutdown_connections 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 -- 172.21.15.45:0/3327501514 >> 172.21.15.45:0/3327501514 conn(0x7f41180a4a10 msgr2=0x7f41180a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 -- 172.21.15.45:0/3327501514 shutdown_connections 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.708+0000 7f411edd8640 1 -- 172.21.15.45:0/3327501514 wait complete. 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: build_initial_monmap 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: monmap: 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 librados: starting msgr at 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 librados: starting objecter 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 Processor -- start 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 -- start start 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 librados: setting wanted keys 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 librados: calling monclient init 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: init 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient(hunting): picked mon.noname-b con 0x7f411809e850 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient(hunting): picked mon.noname-c con 0x7f4118137b40 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient(hunting): picked mon.noname-a con 0x7f411813e170 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f411813d600 con 0x7f411813e170 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f411811e600 con 0x7f411809e850 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f411813b860 con 0x7f4118137b40 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient(hunting): authenticate will time out at 2013.364914s 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411d375640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f411813e170 0x7f41181d4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44634/0 (socket says 172.21.15.45:44634) 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f4117fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4118137b40 0x7f411813a840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50284/0 (socket says 172.21.15.45:50284) 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411d375640 1 -- 172.21.15.45:0/646808356 learned_addr learned my addr 172.21.15.45:0/646808356 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 client.?.objecter ms_handle_connect 0x7f411813e170 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 client.?.objecter ms_handle_connect 0x7f4118137b40 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 715532845 0 0) 0x7f411813d600 con 0x7f411813e170 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): my global_id is 174123 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4104004ec0 con 0x7f411813e170 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1242164521 0 0) 0x7f411813b860 con 0x7f4118137b40 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): my global_id is 144452 2024-09-16T03:22:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f411813d600 con 0x7f4118137b40 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 client.?.objecter ms_handle_connect 0x7f411809e850 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3214466879 0 0) 0x7f4104004ec0 con 0x7f411813e170 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f411813b860 con 0x7f411813e170 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4110001340 con 0x7f411813e170 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): dump: 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1439784885 0 0) 0x7f411813d600 con 0x7f4118137b40 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4104004ec0 con 0x7f4118137b40 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f410c001820 con 0x7f4118137b40 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): dump: 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249366772 0 0) 0x7f411811e600 con 0x7f411809e850 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _negotiate 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient(hunting): my global_id is 164227 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f411813d600 con 0x7f411809e850 2024-09-16T03:22:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3688681604 0 0) 0x7f411813b860 con 0x7f411813e170 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 >> v1:172.21.15.45:6790/0 conn(0x7f4118137b40 legacy=0x7f411813a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 >> v1:172.21.15.46:6789/0 conn(0x7f411809e850 legacy=0x7f411813a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient: _finish_hunting 0 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 monclient: found mon.a 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411809f100 con 0x7f411813e170 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient: _finish_auth 0 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient: _check_auth_tickets 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 5 monclient: authenticate success, global_id 174123 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: _renew_subs 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f411813d840 con 0x7f411813e170 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 client.174123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: _renew_subs 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f411813af50 con 0x7f411813e170 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 librados: init done 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41100047f0 con 0x7f411813e170 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4110004b40 con 0x7f411813e170 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f4116ffd640 4 set_mon_vals no callback set 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 monclient: dump: 2024-09-16T03:22:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4110010bb0 con 0x7f411813e170 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f8005090 con 0x7f411813e170 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(694..694 src has 99..694) ==== 5930+0+0 (unknown 2599890563 0 0) 0x7f411004b930 con 0x7f411813e170 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 client.174123.objecter ms_dispatch 0x7f4118000ba0 osd_map(694..694 src has 99..694) 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 3 client.174123.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 3 client.174123.objecter handle_osd_map decoding full epoch 694 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 20 client.174123.objecter dump_active .. 0 homeless 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f411cb74640 10 monclient: get_auth_request con 0x7f410403e300 auth_method 0 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.712+0000 7f41157fa640 10 client.174123.objecter ms_handle_connect 0x7f410403e300 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.716+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4110016990 con 0x7f411813e170 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.716+0000 7f41157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.716+0000 7f41157fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f411edd8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:22:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f411edd8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:22:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f411edd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f40f8005b90 con 0x7f411813e170 2024-09-16T03:22:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f41157fa640 1 -- 172.21.15.45:0/646808356 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f4110016290 con 0x7f411813e170 2024-09-16T03:22:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f41157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:22:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f41157fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:02.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=9,kv=13,logm=49} crc {auth=2996517887,config=1097430493,health=981150289,kv=505194352,logm=2774963844}) 2024-09-16T03:22:02.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=14,logm=47} crc {auth=3163134167,config=1097430493,health=1265858556,kv=3374818887,logm=2413759027}) 2024-09-16T03:22:02.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=9,kv=13,logm=49} crc {auth=2996517887,config=1097430493,health=981150289,kv=505194352,logm=2774963844}) 2024-09-16T03:22:02.962 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.956+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=10,kv=12,logm=50} crc {auth=2248199058,config=1097430493,health=1816627330,kv=4019763501,logm=2610708558}) 2024-09-16T03:22:02.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 10 librados: watch_flush enter 2024-09-16T03:22:02.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 10 librados: watch_flush exit 2024-09-16T03:22:02.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 20 client.174123.objecter shutdown clearing up homeless session... 2024-09-16T03:22:02.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 10 client.174123.objecter successfully canceled tick 2024-09-16T03:22:02.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 >> v1:172.21.15.46:6812/149360280 conn(0x7f410403e300 legacy=0x7f41040407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 10 monclient: shutdown 2024-09-16T03:22:02.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:02.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 >> v1:172.21.15.45:6789/0 conn(0x7f411813e170 legacy=0x7f41181d4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:02.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 shutdown_connections 2024-09-16T03:22:02.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 >> 172.21.15.45:0/646808356 conn(0x7f41180a4a10 msgr2=0x7f41180a1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:02.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 shutdown_connections 2024-09-16T03:22:02.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 1 -- 172.21.15.45:0/646808356 wait complete. 2024-09-16T03:22:02.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f411edd8640 1 librados: shutdown 2024-09-16T03:22:02.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5824533}) 2024-09-16T03:22:02.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2214009246}) 2024-09-16T03:22:02.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5824533}) 2024-09-16T03:22:02.965 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.960+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3105396291}) 2024-09-16T03:22:02.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830052358}) 2024-09-16T03:22:02.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182196262}) 2024-09-16T03:22:02.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830052358}) 2024-09-16T03:22:02.967 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=173709441}) 2024-09-16T03:22:02.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791633026}) 2024-09-16T03:22:02.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1791245578}) 2024-09-16T03:22:02.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791633026}) 2024-09-16T03:22:02.972 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.964+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910363170}) 2024-09-16T03:22:02.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.972+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.972+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7153419}) 2024-09-16T03:22:02.975 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.972+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3950987259}) 2024-09-16T03:22:02.976 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.972+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.976 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.972+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7153419}) 2024-09-16T03:22:02.976 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.972+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=138998751}) 2024-09-16T03:22:02.983 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:22:02.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.980+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.984+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050238228}) 2024-09-16T03:22:02.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.984+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505959563}) 2024-09-16T03:22:02.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.984+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.984+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050238228}) 2024-09-16T03:22:02.986 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.984+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246429623}) 2024-09-16T03:22:02.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350032401}) 2024-09-16T03:22:02.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253463827}) 2024-09-16T03:22:02.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350032401}) 2024-09-16T03:22:02.996 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2095484722}) 2024-09-16T03:22:02.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229067220}) 2024-09-16T03:22:02.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2235299650}) 2024-09-16T03:22:02.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:02.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229067220}) 2024-09-16T03:22:02.998 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.992+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2479170848}) 2024-09-16T03:22:03.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.996+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.996+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835731172}) 2024-09-16T03:22:03.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.996+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2847302591}) 2024-09-16T03:22:03.000 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.996+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.001 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.996+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835731172}) 2024-09-16T03:22:03.001 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.996+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3454332818}) 2024-09-16T03:22:03.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.996+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.996+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1313211712}) 2024-09-16T03:22:03.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:02.996+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2721608476}) 2024-09-16T03:22:03.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1313211712}) 2024-09-16T03:22:03.002 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2061098262}) 2024-09-16T03:22:03.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063690840}) 2024-09-16T03:22:03.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=766948433}) 2024-09-16T03:22:03.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063690840}) 2024-09-16T03:22:03.004 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647420502}) 2024-09-16T03:22:03.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3158385941}) 2024-09-16T03:22:03.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387567669}) 2024-09-16T03:22:03.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3158385941}) 2024-09-16T03:22:03.006 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.000+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563345970}) 2024-09-16T03:22:03.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.004+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.004+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411612179}) 2024-09-16T03:22:03.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.004+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=217548337}) 2024-09-16T03:22:03.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.004+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.004+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411612179}) 2024-09-16T03:22:03.008 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.004+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=544648572}) 2024-09-16T03:22:03.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065641360}) 2024-09-16T03:22:03.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3992452855}) 2024-09-16T03:22:03.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065641360}) 2024-09-16T03:22:03.010 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965342994}) 2024-09-16T03:22:03.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1178313734}) 2024-09-16T03:22:03.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801005173}) 2024-09-16T03:22:03.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1178313734}) 2024-09-16T03:22:03.012 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.008+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1696987236}) 2024-09-16T03:22:03.014 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.014 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492494139}) 2024-09-16T03:22:03.014 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=537096030}) 2024-09-16T03:22:03.014 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.014 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492494139}) 2024-09-16T03:22:03.015 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3069590721}) 2024-09-16T03:22:03.016 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076489103}) 2024-09-16T03:22:03.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2843227850}) 2024-09-16T03:22:03.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076489103}) 2024-09-16T03:22:03.017 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.012+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776851085}) 2024-09-16T03:22:03.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32516746}) 2024-09-16T03:22:03.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=782187951}) 2024-09-16T03:22:03.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32516746}) 2024-09-16T03:22:03.019 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3551780251}) 2024-09-16T03:22:03.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828270717}) 2024-09-16T03:22:03.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2408728318}) 2024-09-16T03:22:03.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828270717}) 2024-09-16T03:22:03.021 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.016+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4148491484}) 2024-09-16T03:22:03.023 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.024 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412692677}) 2024-09-16T03:22:03.024 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389205216}) 2024-09-16T03:22:03.024 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.024 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412692677}) 2024-09-16T03:22:03.024 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2566721254}) 2024-09-16T03:22:03.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1509028211}) 2024-09-16T03:22:03.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4008497537}) 2024-09-16T03:22:03.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1509028211}) 2024-09-16T03:22:03.026 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.020+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=155155779}) 2024-09-16T03:22:03.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.024+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.024+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2054549244}) 2024-09-16T03:22:03.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.024+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3443159942}) 2024-09-16T03:22:03.028 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.024+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.029 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.024+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2054549244}) 2024-09-16T03:22:03.029 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.024+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568105264}) 2024-09-16T03:22:03.030 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.030 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3653835275}) 2024-09-16T03:22:03.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=896244334}) 2024-09-16T03:22:03.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3653835275}) 2024-09-16T03:22:03.031 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4057724820}) 2024-09-16T03:22:03.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245960649}) 2024-09-16T03:22:03.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3435717453}) 2024-09-16T03:22:03.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245960649}) 2024-09-16T03:22:03.033 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.028+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3739844682}) 2024-09-16T03:22:03.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834300808}) 2024-09-16T03:22:03.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521772250}) 2024-09-16T03:22:03.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834300808}) 2024-09-16T03:22:03.035 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798119506}) 2024-09-16T03:22:03.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968667658}) 2024-09-16T03:22:03.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1681681483}) 2024-09-16T03:22:03.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.037 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968667658}) 2024-09-16T03:22:03.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.032+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=869533226}) 2024-09-16T03:22:03.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.036+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.036+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3794316807}) 2024-09-16T03:22:03.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.036+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350865807}) 2024-09-16T03:22:03.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.036+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.036+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3794316807}) 2024-09-16T03:22:03.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.036+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3992405975}) 2024-09-16T03:22:03.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1779851148}) 2024-09-16T03:22:03.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=990086545}) 2024-09-16T03:22:03.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1779851148}) 2024-09-16T03:22:03.042 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2352531769}) 2024-09-16T03:22:03.044 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.044 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1700883960}) 2024-09-16T03:22:03.044 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801310046}) 2024-09-16T03:22:03.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1700883960}) 2024-09-16T03:22:03.045 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.040+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4058561151}) 2024-09-16T03:22:03.046 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661289068}) 2024-09-16T03:22:03.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1170212482}) 2024-09-16T03:22:03.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661289068}) 2024-09-16T03:22:03.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160660788}) 2024-09-16T03:22:03.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1095607896}) 2024-09-16T03:22:03.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2434193556}) 2024-09-16T03:22:03.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1095607896}) 2024-09-16T03:22:03.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.044+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1002177992}) 2024-09-16T03:22:03.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2173822446}) 2024-09-16T03:22:03.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3243332092}) 2024-09-16T03:22:03.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2173822446}) 2024-09-16T03:22:03.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1764514949}) 2024-09-16T03:22:03.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.053 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3449095689}) 2024-09-16T03:22:03.054 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=949151778}) 2024-09-16T03:22:03.054 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.054 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3449095689}) 2024-09-16T03:22:03.054 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.048+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=282213988}) 2024-09-16T03:22:03.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.052+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.056 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.052+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086152494}) 2024-09-16T03:22:03.056 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.052+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=57333205}) 2024-09-16T03:22:03.056 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.052+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.056 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.052+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086152494}) 2024-09-16T03:22:03.056 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.052+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3006621447}) 2024-09-16T03:22:03.058 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.052+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.058 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639937624}) 2024-09-16T03:22:03.058 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782981845}) 2024-09-16T03:22:03.058 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.058 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639937624}) 2024-09-16T03:22:03.058 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=861073508}) 2024-09-16T03:22:03.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859437093}) 2024-09-16T03:22:03.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1465238957}) 2024-09-16T03:22:03.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.060 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859437093}) 2024-09-16T03:22:03.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.056+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2520080343}) 2024-09-16T03:22:03.062 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.062 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251120596}) 2024-09-16T03:22:03.062 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=168773206}) 2024-09-16T03:22:03.062 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.063 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251120596}) 2024-09-16T03:22:03.063 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3011599110}) 2024-09-16T03:22:03.064 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.064 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2061712372}) 2024-09-16T03:22:03.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71966168}) 2024-09-16T03:22:03.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2061712372}) 2024-09-16T03:22:03.065 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.060+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3443406619}) 2024-09-16T03:22:03.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1437513921}) 2024-09-16T03:22:03.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2418885578}) 2024-09-16T03:22:03.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1437513921}) 2024-09-16T03:22:03.067 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3899360136}) 2024-09-16T03:22:03.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024664885}) 2024-09-16T03:22:03.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=352410548}) 2024-09-16T03:22:03.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024664885}) 2024-09-16T03:22:03.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.064+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2498885176}) 2024-09-16T03:22:03.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555102359}) 2024-09-16T03:22:03.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528106509}) 2024-09-16T03:22:03.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555102359}) 2024-09-16T03:22:03.071 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288047811}) 2024-09-16T03:22:03.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749563197}) 2024-09-16T03:22:03.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879594967}) 2024-09-16T03:22:03.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749563197}) 2024-09-16T03:22:03.073 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.068+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4228901610}) 2024-09-16T03:22:03.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.072+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.076 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.072+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641496943}) 2024-09-16T03:22:03.076 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.072+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3951202287}) 2024-09-16T03:22:03.076 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.072+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.076 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.072+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641496943}) 2024-09-16T03:22:03.076 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.072+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2890120309}) 2024-09-16T03:22:03.078 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.078 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042109583}) 2024-09-16T03:22:03.078 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751095786}) 2024-09-16T03:22:03.078 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.078 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042109583}) 2024-09-16T03:22:03.078 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2760604545}) 2024-09-16T03:22:03.080 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.080 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779226196}) 2024-09-16T03:22:03.080 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1342942008}) 2024-09-16T03:22:03.080 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.080 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779226196}) 2024-09-16T03:22:03.080 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.076+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3081359590}) 2024-09-16T03:22:03.082 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.082 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769149664}) 2024-09-16T03:22:03.082 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469342932}) 2024-09-16T03:22:03.083 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.083 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769149664}) 2024-09-16T03:22:03.083 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=249412174}) 2024-09-16T03:22:03.085 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.085 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127364711}) 2024-09-16T03:22:03.085 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3121346616}) 2024-09-16T03:22:03.085 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.085 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127364711}) 2024-09-16T03:22:03.085 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.080+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784136649}) 2024-09-16T03:22:03.087 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.087 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936915481}) 2024-09-16T03:22:03.087 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1399644707}) 2024-09-16T03:22:03.087 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.087 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936915481}) 2024-09-16T03:22:03.087 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=99275663}) 2024-09-16T03:22:03.089 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.089 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648414868}) 2024-09-16T03:22:03.089 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3214935330}) 2024-09-16T03:22:03.089 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.089 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648414868}) 2024-09-16T03:22:03.089 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.084+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51493615}) 2024-09-16T03:22:03.091 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.088+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.091 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.088+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629948973}) 2024-09-16T03:22:03.092 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.088+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2019491551}) 2024-09-16T03:22:03.092 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.088+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.092 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.088+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629948973}) 2024-09-16T03:22:03.092 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.088+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=121071699}) 2024-09-16T03:22:03.094 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.088+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.094 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432289658}) 2024-09-16T03:22:03.094 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2160065935}) 2024-09-16T03:22:03.094 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.094 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432289658}) 2024-09-16T03:22:03.094 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2096080024}) 2024-09-16T03:22:03.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684953574}) 2024-09-16T03:22:03.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848759332}) 2024-09-16T03:22:03.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684953574}) 2024-09-16T03:22:03.096 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.092+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2348146023}) 2024-09-16T03:22:03.098 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.098 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2315429807}) 2024-09-16T03:22:03.098 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603988045}) 2024-09-16T03:22:03.098 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.098 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2315429807}) 2024-09-16T03:22:03.098 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392672078}) 2024-09-16T03:22:03.101 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.101 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460577706}) 2024-09-16T03:22:03.101 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335126523}) 2024-09-16T03:22:03.101 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.101 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460577706}) 2024-09-16T03:22:03.101 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.096+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907945533}) 2024-09-16T03:22:03.103 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.103 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1811685491}) 2024-09-16T03:22:03.103 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86900530}) 2024-09-16T03:22:03.103 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.103 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1811685491}) 2024-09-16T03:22:03.103 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4090277231}) 2024-09-16T03:22:03.105 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.105 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983187216}) 2024-09-16T03:22:03.105 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622193188}) 2024-09-16T03:22:03.105 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.105 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983187216}) 2024-09-16T03:22:03.106 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.100+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145482804}) 2024-09-16T03:22:03.107 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773658592}) 2024-09-16T03:22:03.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3041676885}) 2024-09-16T03:22:03.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773658592}) 2024-09-16T03:22:03.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3828845754}) 2024-09-16T03:22:03.110 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.110 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376846591}) 2024-09-16T03:22:03.110 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1426855769}) 2024-09-16T03:22:03.110 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.110 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.104+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376846591}) 2024-09-16T03:22:03.110 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.108+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=710927565}) 2024-09-16T03:22:03.112 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.108+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.112 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.108+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4026868178}) 2024-09-16T03:22:03.112 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.108+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2709962063}) 2024-09-16T03:22:03.112 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.108+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.112 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.108+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4026868178}) 2024-09-16T03:22:03.112 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.108+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2859791612}) 2024-09-16T03:22:03.114 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.108+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.114 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.108+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2929671737}) 2024-09-16T03:22:03.114 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1077634716}) 2024-09-16T03:22:03.114 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.114 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2929671737}) 2024-09-16T03:22:03.114 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=475892942}) 2024-09-16T03:22:03.116 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.116 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963827920}) 2024-09-16T03:22:03.116 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2482204706}) 2024-09-16T03:22:03.116 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.116 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963827920}) 2024-09-16T03:22:03.117 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.112+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=741648415}) 2024-09-16T03:22:03.118 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.118 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017822263}) 2024-09-16T03:22:03.118 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3714449699}) 2024-09-16T03:22:03.118 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.119 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017822263}) 2024-09-16T03:22:03.119 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=260331876}) 2024-09-16T03:22:03.120 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=393268375}) 2024-09-16T03:22:03.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787328765}) 2024-09-16T03:22:03.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=393268375}) 2024-09-16T03:22:03.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.116+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=485197667}) 2024-09-16T03:22:03.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=877896364}) 2024-09-16T03:22:03.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1136014449}) 2024-09-16T03:22:03.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=877896364}) 2024-09-16T03:22:03.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3659954994}) 2024-09-16T03:22:03.125 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.125 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169426215}) 2024-09-16T03:22:03.125 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855249741}) 2024-09-16T03:22:03.125 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.125 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169426215}) 2024-09-16T03:22:03.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.120+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1291791994}) 2024-09-16T03:22:03.127 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.127 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209375469}) 2024-09-16T03:22:03.127 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=773359651}) 2024-09-16T03:22:03.127 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209375469}) 2024-09-16T03:22:03.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2076516911}) 2024-09-16T03:22:03.129 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.129 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303939588}) 2024-09-16T03:22:03.129 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3803785784}) 2024-09-16T03:22:03.130 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.130 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303939588}) 2024-09-16T03:22:03.130 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.124+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2552897601}) 2024-09-16T03:22:03.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.128+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.128+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885200851}) 2024-09-16T03:22:03.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.128+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=744437133}) 2024-09-16T03:22:03.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.128+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.128+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885200851}) 2024-09-16T03:22:03.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.128+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498699120}) 2024-09-16T03:22:03.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3887077832}) 2024-09-16T03:22:03.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2769680936}) 2024-09-16T03:22:03.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3887077832}) 2024-09-16T03:22:03.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2894218571}) 2024-09-16T03:22:03.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962665946}) 2024-09-16T03:22:03.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1567082818}) 2024-09-16T03:22:03.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962665946}) 2024-09-16T03:22:03.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.132+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232996423}) 2024-09-16T03:22:03.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42369418}) 2024-09-16T03:22:03.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2016518619}) 2024-09-16T03:22:03.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42369418}) 2024-09-16T03:22:03.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2900459621}) 2024-09-16T03:22:03.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=2} crc {logm=1598329733,mdsmap=2287870078,mgr=3494632410,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1425958776,mon_config_key=2994965654,monmap=127340147}) 2024-09-16T03:22:03.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=18} crc {logm=3348553068,mdsmap=2287870078,mgr=3993191508,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=3588556138,mon_config_key=1985398318}) 2024-09-16T03:22:03.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=2} crc {logm=1598329733,mdsmap=2287870078,mgr=3494632410,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1425958776,mon_config_key=2994965654,monmap=127340147}) 2024-09-16T03:22:03.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.136+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=61,mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=18} crc {logm=3118763068,mdsmap=3140514009,mgr=3873834454,mgr_metadata=127159686,mgrstat=885591896,mon_config_key=2518427641}) 2024-09-16T03:22:03.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.140+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.140+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=51} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=3965956840,osd_snap=230152691,osdmap=1939511221}) 2024-09-16T03:22:03.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.140+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=51} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=3965956840,osd_snap=727228197,osdmap=3922557742}) 2024-09-16T03:22:03.144 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.140+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.144 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.140+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=51} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=3965956840,osd_snap=230152691,osdmap=1939511221}) 2024-09-16T03:22:03.144 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.140+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=49} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=2004720307,osd_pg_creating=3965956840,osd_snap=1745428576,osdmap=3854523090}) 2024-09-16T03:22:03.146 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.146 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2962315056}) 2024-09-16T03:22:03.146 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1894688569}) 2024-09-16T03:22:03.146 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.146 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2962315056}) 2024-09-16T03:22:03.147 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2037545621}) 2024-09-16T03:22:03.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2193368864}) 2024-09-16T03:22:03.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=786260393}) 2024-09-16T03:22:03.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2193368864}) 2024-09-16T03:22:03.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.144+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2789210666}) 2024-09-16T03:22:03.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.148+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.148+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=929379195}) 2024-09-16T03:22:03.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.148+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3110507154}) 2024-09-16T03:22:03.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.148+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.148+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=929379195}) 2024-09-16T03:22:03.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.148+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3871769215}) 2024-09-16T03:22:03.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1236690776}) 2024-09-16T03:22:03.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3848681622}) 2024-09-16T03:22:03.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1236690776}) 2024-09-16T03:22:03.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1926372868}) 2024-09-16T03:22:03.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2548244630}) 2024-09-16T03:22:03.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2291236879}) 2024-09-16T03:22:03.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2548244630}) 2024-09-16T03:22:03.157 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.152+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2315033394}) 2024-09-16T03:22:03.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.156+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.156+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=378389822}) 2024-09-16T03:22:03.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.156+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=641839148}) 2024-09-16T03:22:03.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.156+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.156+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=378389822}) 2024-09-16T03:22:03.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.156+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=779222111}) 2024-09-16T03:22:03.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.160+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.160+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1143575204}) 2024-09-16T03:22:03.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.160+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4022252779}) 2024-09-16T03:22:03.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.160+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.166 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.160+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1143575204}) 2024-09-16T03:22:03.166 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.160+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1344305}) 2024-09-16T03:22:03.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.168+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.171 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.168+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3267849520}) 2024-09-16T03:22:03.171 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.168+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=959846638}) 2024-09-16T03:22:03.171 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.168+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.171 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.168+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3267849520}) 2024-09-16T03:22:03.171 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.168+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=139866235}) 2024-09-16T03:22:03.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.180+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.180+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=15} crc {osdmap=240083220}) 2024-09-16T03:22:03.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.180+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=13} crc {osdmap=1014309818}) 2024-09-16T03:22:03.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.180+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:22:03.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.180+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=15} crc {osdmap=240083220}) 2024-09-16T03:22:03.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:22:03.180+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=14} crc {osdmap=141792412}) 2024-09-16T03:22:03.983 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:22:03.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:22:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.116+0000 7f51604c3640 10 monclient: get_monmap_and_config 2024-09-16T03:22:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.116+0000 7f51604c3640 10 monclient: build_initial_monmap 2024-09-16T03:22:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.116+0000 7f51604c3640 10 monclient: monmap: 2024-09-16T03:22:04.120 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:04.122587+0000 2024-09-16T03:22:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:04.122587+0000 2024-09-16T03:22:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:04.121 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:04.121 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.116+0000 7f51604c3640 1 Processor -- start 2024-09-16T03:22:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.116+0000 7f51604c3640 1 -- start start 2024-09-16T03:22:04.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.116+0000 7f51604c3640 10 monclient: init 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): picked mon.noname-b con 0x7f5158137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): picked mon.noname-c con 0x7f5158138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): picked mon.noname-a con 0x7f515813e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51580a4f90 con 0x7f515813e2a0 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51580a4cb0 con 0x7f5158138100 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515809dc70 con 0x7f5158137d20 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): authenticate will time out at 2014.772629s 2024-09-16T03:22:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515e25f640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5158137d20 0x7f51580a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50294/0 (socket says 172.21.15.45:50294) 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515e25f640 1 -- 172.21.15.45:0/2319648538 learned_addr learned my addr 172.21.15.45:0/2319648538 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815118731 0 0) 0x7f51580a4f90 con 0x7f515813e2a0 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient(hunting): my global_id is 174126 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5144003530 con 0x7f515813e2a0 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3981624616 0 0) 0x7f515809dc70 con 0x7f5158137d20 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient(hunting): my global_id is 144455 2024-09-16T03:22:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51580a4f90 con 0x7f5158137d20 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3537452352 0 0) 0x7f51580a4f90 con 0x7f5158137d20 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 >> v1:172.21.15.46:6789/0 conn(0x7f5158138100 legacy=0x7f515813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 >> v1:172.21.15.45:6789/0 conn(0x7f515813e2a0 legacy=0x7f5158140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: _finish_hunting 0 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 monclient: found mon.noname-b 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515809cc50 con 0x7f5158137d20 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: _finish_auth 0 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: _check_auth_tickets 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5154001680 con 0x7f5158137d20 2024-09-16T03:22:04.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: dump: 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 5 monclient: authenticate success, global_id 144455 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f51540018a0 con 0x7f5158137d20 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 1 -- 172.21.15.45:0/2319648538 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5154001cb0 con 0x7f5158137d20 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 4 set_mon_vals no callback set 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515d25d640 10 monclient: dump: 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.129 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient: shutdown 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- 172.21.15.45:0/2319648538 >> v1:172.21.15.45:6790/0 conn(0x7f5158137d20 legacy=0x7f51580a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- 172.21.15.45:0/2319648538 shutdown_connections 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- 172.21.15.45:0/2319648538 >> 172.21.15.45:0/2319648538 conn(0x7f51580aa110 msgr2=0x7f51580aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- 172.21.15.45:0/2319648538 shutdown_connections 2024-09-16T03:22:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- 172.21.15.45:0/2319648538 wait complete. 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient: build_initial_monmap 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient: monmap: 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 librados: starting msgr at 2024-09-16T03:22:04.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 librados: starting objecter 2024-09-16T03:22:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 Processor -- start 2024-09-16T03:22:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- start start 2024-09-16T03:22:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 librados: setting wanted keys 2024-09-16T03:22:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 librados: calling monclient init 2024-09-16T03:22:04.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient: init 2024-09-16T03:22:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:22:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): picked mon.noname-c con 0x7f5158137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): picked mon.noname-a con 0x7f5158138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:04.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): picked mon.noname-b con 0x7f515813e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5158141b30 con 0x7f5158138100 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51580a1300 con 0x7f515813e2a0 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515813fe10 con 0x7f5158137d20 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f51604c3640 10 monclient(hunting): authenticate will time out at 2014.775731s 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515e25f640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5158137d20 0x7f515813e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50298/0 (socket says 172.21.15.45:50298) 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515e25f640 1 -- 172.21.15.45:0/2492871637 learned_addr learned my addr 172.21.15.45:0/2492871637 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f515da5e640 1 --1- 172.21.15.45:0/2492871637 >> v1:172.21.15.45:6789/0 conn(0x7f5158138100 0x7f515813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44652/0 (socket says 172.21.15.45:44652) 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f514effd640 10 client.?.objecter ms_handle_connect 0x7f5158137d20 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f514effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f514effd640 10 client.?.objecter ms_handle_connect 0x7f5158138100 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.120+0000 7f514effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249150712 0 0) 0x7f515813fe10 con 0x7f5158137d20 2024-09-16T03:22:04.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): my global_id is 144458 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f513c004ec0 con 0x7f5158137d20 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132955601 0 0) 0x7f5158141b30 con 0x7f5158138100 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): my global_id is 174129 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f515813fe10 con 0x7f5158138100 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 client.?.objecter ms_handle_connect 0x7f515813e2a0 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2687481851 0 0) 0x7f513c004ec0 con 0x7f5158137d20 2024-09-16T03:22:04.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5158141b30 con 0x7f5158137d20 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5154001cf0 con 0x7f5158137d20 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): dump: 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1211369389 0 0) 0x7f515813fe10 con 0x7f5158138100 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f513c004ec0 con 0x7f5158138100 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3511169122 0 0) 0x7f51580a1300 con 0x7f515813e2a0 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): my global_id is 164233 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f515813fe10 con 0x7f515813e2a0 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f51480018c0 con 0x7f5158138100 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient(hunting): dump: 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.140 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 601814995 0 0) 0x7f5158141b30 con 0x7f5158137d20 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 >> v1:172.21.15.46:6789/0 conn(0x7f515813e2a0 legacy=0x7f51581d4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 >> v1:172.21.15.45:6789/0 conn(0x7f5158138100 legacy=0x7f515813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient: _finish_hunting 0 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 monclient: found mon.c 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:04.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5158138a60 con 0x7f5158137d20 2024-09-16T03:22:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient: _finish_auth 0 2024-09-16T03:22:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient: _check_auth_tickets 2024-09-16T03:22:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 5 monclient: authenticate success, global_id 144458 2024-09-16T03:22:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 10 monclient: _renew_subs 2024-09-16T03:22:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f51581376e0 con 0x7f5158137d20 2024-09-16T03:22:04.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5154004b20 con 0x7f5158137d20 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 10 client.144458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5154004ef0 con 0x7f5158137d20 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 10 monclient: _renew_subs 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f515813f4a0 con 0x7f5158137d20 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 1 librados: init done 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f515ca5c640 4 set_mon_vals no callback set 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 monclient: dump: 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.143 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:04.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51580008e0 con 0x7f5158137d20 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f5154010f60 con 0x7f5158137d20 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(696..696 src has 99..696) ==== 5917+0+0 (unknown 4201109949 0 0) 0x7f515404ba00 con 0x7f5158137d20 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 client.144458.objecter ms_dispatch 0x7f5158000ba0 osd_map(696..696 src has 99..696) 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 3 client.144458.objecter handle_osd_map got epochs [696,696] > 0 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 3 client.144458.objecter handle_osd_map decoding full epoch 696 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 20 client.144458.objecter dump_active .. 0 homeless 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f515da5e640 10 monclient: get_auth_request con 0x7f513c03e240 auth_method 0 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.124+0000 7f514effd640 10 client.144458.objecter ms_handle_connect 0x7f513c03e240 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.128+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5154016b30 con 0x7f5158137d20 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.128+0000 7f514effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:04.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.128+0000 7f514effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:04.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.364+0000 7f51604c3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:22:04.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.364+0000 7f51604c3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:04.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.364+0000 7f51604c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:22:04.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.364+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f51580770b0 con 0x7f5158137d20 2024-09-16T03:22:04.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f514effd640 1 -- 172.21.15.45:0/2492871637 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 1776753124) 0x7f5154016430 con 0x7f5158137d20 2024-09-16T03:22:04.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f514effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:04.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f514effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:04.370 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:22:04.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 10 librados: watch_flush enter 2024-09-16T03:22:04.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 10 librados: watch_flush exit 2024-09-16T03:22:04.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 20 client.144458.objecter shutdown clearing up homeless session... 2024-09-16T03:22:04.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 10 client.144458.objecter successfully canceled tick 2024-09-16T03:22:04.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 >> v1:172.21.15.46:6812/149360280 conn(0x7f513c03e240 legacy=0x7f513c040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 10 monclient: shutdown 2024-09-16T03:22:04.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:04.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 >> v1:172.21.15.45:6790/0 conn(0x7f5158137d20 legacy=0x7f515813e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 shutdown_connections 2024-09-16T03:22:04.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 >> 172.21.15.45:0/2492871637 conn(0x7f51580aa110 msgr2=0x7f51580731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:04.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 shutdown_connections 2024-09-16T03:22:04.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 1 -- 172.21.15.45:0/2492871637 wait complete. 2024-09-16T03:22:04.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.368+0000 7f51604c3640 1 librados: shutdown 2024-09-16T03:22:04.390 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:22:04.390 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:22:04.390 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:22:04.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:22:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient: get_monmap_and_config 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient: build_initial_monmap 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient: monmap: 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:04.517928+0000 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:04.517928+0000 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:04.517 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 1 Processor -- start 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 1 -- start start 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient: init 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient(hunting): picked mon.noname-b con 0x7f85f8137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient(hunting): picked mon.noname-c con 0x7f85f8138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient(hunting): picked mon.noname-a con 0x7f85f813e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f80a4f90 con 0x7f85f813e2a0 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f80a4cb0 con 0x7f85f8138100 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f809dc70 con 0x7f85f8137d20 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:04.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85fd5b6640 10 monclient(hunting): authenticate will time out at 2015.167682s 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85f6ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f85f8137d20 0x7f85f80a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50300/0 (socket says 172.21.15.45:50300) 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85f6ffd640 1 -- 172.21.15.45:0/2808256274 learned_addr learned my addr 172.21.15.45:0/2808256274 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.512+0000 7f85f77fe640 1 --1- 172.21.15.45:0/2808256274 >> v1:172.21.15.45:6789/0 conn(0x7f85f813e2a0 0x7f85f8140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44662/0 (socket says 172.21.15.45:44662) 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557011119 0 0) 0x7f85f809dc70 con 0x7f85f8137d20 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient(hunting): my global_id is 144461 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85e4003530 con 0x7f85f8137d20 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408590132 0 0) 0x7f85f80a4f90 con 0x7f85f813e2a0 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient(hunting): my global_id is 174132 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f809dc70 con 0x7f85f813e2a0 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3224801253 0 0) 0x7f85f809dc70 con 0x7f85f813e2a0 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 >> v1:172.21.15.45:6790/0 conn(0x7f85f8137d20 legacy=0x7f85f80a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 >> v1:172.21.15.46:6789/0 conn(0x7f85f8138100 legacy=0x7f85f813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: _finish_hunting 0 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 monclient: found mon.noname-a 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f809cc50 con 0x7f85f813e2a0 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: _finish_auth 0 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f85f0001a00 con 0x7f85f813e2a0 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 5 monclient: authenticate success, global_id 174132 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: dump: 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.528 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f85f0001c20 con 0x7f85f813e2a0 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 1 -- 172.21.15.45:0/2808256274 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f85f0002070 con 0x7f85f813e2a0 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 4 set_mon_vals no callback set 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f67fc640 10 monclient: dump: 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.529 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: shutdown 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- 172.21.15.45:0/2808256274 >> v1:172.21.15.45:6789/0 conn(0x7f85f813e2a0 legacy=0x7f85f8140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- 172.21.15.45:0/2808256274 shutdown_connections 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- 172.21.15.45:0/2808256274 >> 172.21.15.45:0/2808256274 conn(0x7f85f80aa110 msgr2=0x7f85f80aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- 172.21.15.45:0/2808256274 shutdown_connections 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- 172.21.15.45:0/2808256274 wait complete. 2024-09-16T03:22:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: build_initial_monmap 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: monmap: 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 librados: starting msgr at 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 librados: starting objecter 2024-09-16T03:22:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 Processor -- start 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- start start 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 librados: setting wanted keys 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 librados: calling monclient init 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: init 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient(hunting): picked mon.noname-b con 0x7f85f8137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient(hunting): picked mon.noname-c con 0x7f85f8138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient(hunting): picked mon.noname-a con 0x7f85f813e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f80a3df0 con 0x7f85f813e2a0 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f80a40d0 con 0x7f85f8137d20 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f80a3020 con 0x7f85f8138100 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:04.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient(hunting): authenticate will time out at 2015.170787s 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f77fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f85f813e2a0 0x7f85f81d4d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44664/0 (socket says 172.21.15.45:44664) 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85effff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f85f8138100 0x7f85f80a1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50302/0 (socket says 172.21.15.45:50302) 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85f77fe640 1 -- 172.21.15.45:0/452652763 learned_addr learned my addr 172.21.15.45:0/452652763 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 client.?.objecter ms_handle_connect 0x7f85f8138100 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 client.?.objecter ms_handle_connect 0x7f85f813e2a0 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 client.?.objecter ms_handle_connect 0x7f85f8137d20 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2768732846 0 0) 0x7f85f80a3020 con 0x7f85f8138100 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): my global_id is 144464 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85d8004ec0 con 0x7f85f8138100 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151762452 0 0) 0x7f85f80a3df0 con 0x7f85f813e2a0 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): my global_id is 174135 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f80a3020 con 0x7f85f813e2a0 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753635797 0 0) 0x7f85f80a40d0 con 0x7f85f8137d20 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): my global_id is 164239 2024-09-16T03:22:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f80a3df0 con 0x7f85f8137d20 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2312117770 0 0) 0x7f85f80a3020 con 0x7f85f813e2a0 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85f80a40d0 con 0x7f85f813e2a0 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f85f0001f70 con 0x7f85f813e2a0 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): dump: 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 906662042 0 0) 0x7f85d8004ec0 con 0x7f85f8138100 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85f80a3020 con 0x7f85f8138100 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f85e0001880 con 0x7f85f8138100 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient(hunting): dump: 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4006172973 0 0) 0x7f85f80a40d0 con 0x7f85f813e2a0 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 >> v1:172.21.15.45:6790/0 conn(0x7f85f8138100 legacy=0x7f85f80a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 >> v1:172.21.15.46:6789/0 conn(0x7f85f8137d20 legacy=0x7f85f80a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 monclient: found mon.a 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f8137670 con 0x7f85f813e2a0 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient: _finish_auth 0 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 5 monclient: authenticate success, global_id 174135 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: _renew_subs 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f85f8140e20 con 0x7f85f813e2a0 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 client.174135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: _renew_subs 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.516+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f85f81d6440 con 0x7f85f813e2a0 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85fd5b6640 1 librados: init done 2024-09-16T03:22:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f85f0001b80 con 0x7f85f813e2a0 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f85f00050e0 con 0x7f85f813e2a0 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85f57fa640 4 set_mon_vals no callback set 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient: dump: 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85fd5b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f85dc0065d0 con 0x7f85dc002870 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 client.174135.objecter ms_handle_connect 0x7f85dc002870 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 client.174135.objecter resend_mon_ops 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f85f0011150 con 0x7f85f813e2a0 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(696..696 src has 99..696) ==== 5917+0+0 (unknown 4201109949 0 0) 0x7f85f004bca0 con 0x7f85f813e2a0 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 client.174135.objecter ms_dispatch 0x7f85f8000ba0 osd_map(696..696 src has 99..696) 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 3 client.174135.objecter handle_osd_map got epochs [696,696] > 0 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 3 client.174135.objecter handle_osd_map decoding full epoch 696 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 20 client.174135.objecter dump_active .. 0 homeless 2024-09-16T03:22:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363605984 0 0) 0x7f85dc0065d0 con 0x7f85dc002870 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient(hunting): my global_id is 174138 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f80a40d0 con 0x7f85dc002870 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85effff640 10 monclient: get_auth_request con 0x7f85d803e150 auth_method 0 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 873912937 0 0) 0x7f85f80a40d0 con 0x7f85dc002870 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85dc0065d0 con 0x7f85dc002870 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f85e80010f0 con 0x7f85dc002870 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient: dump: 2024-09-16T03:22:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4133479313 0 0) 0x7f85dc0065d0 con 0x7f85dc002870 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85dc00a170 con 0x7f85dc002870 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 client.174135.objecter ms_handle_connect 0x7f85d803e150 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f85e8001310 con 0x7f85dc002870 2024-09-16T03:22:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:04.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:04.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 >> v1:172.21.15.45:6789/0 conn(0x7f85dc002870 legacy=0x7f85dc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.520+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 >> v1:172.21.15.45:6789/0 conn(0x7f85dc002870 legacy=0x7f85dc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:04.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f85dc009e80 con 0x7f85dc009140 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 client.174135.objecter ms_handle_connect 0x7f85dc009140 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 client.174135.objecter resend_mon_ops 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 70825628 0 0) 0x7f85dc009e80 con 0x7f85dc009140 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 monclient(hunting): my global_id is 174141 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85dc0065d0 con 0x7f85dc009140 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2676660200 0 0) 0x7f85dc0065d0 con 0x7f85dc009140 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85dc009e80 con 0x7f85dc009140 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f85e8004990 con 0x7f85dc009140 2024-09-16T03:22:04.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 monclient: dump: 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1155099716 0 0) 0x7f85dc009e80 con 0x7f85dc009140 2024-09-16T03:22:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f85dc00c420 con 0x7f85dc009140 2024-09-16T03:22:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3968 (unknown 0 0 939721841) 0x7f85e8006b90 con 0x7f85dc009140 2024-09-16T03:22:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:22:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 >> v1:172.21.15.45:6789/0 conn(0x7f85dc009140 legacy=0x7f85dc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85ef7fe640 1 -- 172.21.15.45:0/452652763 >> v1:172.21.15.45:6789/0 conn(0x7f85dc009140 legacy=0x7f85dc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:04.547 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:22:04.547 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:04.547 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:04.547 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 70, 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 2787, 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:22:04.550 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:04.551 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.552 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:22:04.553 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.554 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.555 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.556 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.557 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.558 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:22:04.558 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:22:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 10 librados: watch_flush enter 2024-09-16T03:22:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 10 librados: watch_flush exit 2024-09-16T03:22:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 20 client.174135.objecter shutdown clearing up homeless session... 2024-09-16T03:22:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 10 client.174135.objecter successfully canceled tick 2024-09-16T03:22:04.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 >> v1:172.21.15.46:6812/149360280 conn(0x7f85d803e150 legacy=0x7f85d8040610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 10 monclient: shutdown 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 >> v1:172.21.15.45:6789/0 conn(0x7f85f813e2a0 legacy=0x7f85f81d4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85f77fe640 1 -- 172.21.15.45:0/452652763 reap_dead start 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85f77fe640 1 -- 172.21.15.45:0/452652763 reap_dead start 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 shutdown_connections 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 >> 172.21.15.45:0/452652763 conn(0x7f85f80aa110 msgr2=0x7f85f8130050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 shutdown_connections 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 1 -- 172.21.15.45:0/452652763 wait complete. 2024-09-16T03:22:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.528+0000 7f85fd5b6640 1 librados: shutdown 2024-09-16T03:22:04.561 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:22:04.561 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:22:04.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:22:04.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: get_monmap_and_config 2024-09-16T03:22:04.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: build_initial_monmap 2024-09-16T03:22:04.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: monmap: 2024-09-16T03:22:04.682 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:04.682 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.682 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:04.684271+0000 2024-09-16T03:22:04.682 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:04.684271+0000 2024-09-16T03:22:04.682 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:04.682 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.683 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:04.683 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:04.683 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:04.683 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 Processor -- start 2024-09-16T03:22:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 -- start start 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: init 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient(hunting): picked mon.noname-b con 0x7f59240a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient(hunting): picked mon.noname-a con 0x7f59240a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient(hunting): picked mon.noname-c con 0x7f5924073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f592409d9e0 con 0x7f59240a5810 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f592409dc20 con 0x7f5924073d40 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5924073360 con 0x7f59240a4d20 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient(hunting): authenticate will time out at 2015.333627s 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592259c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f59240a4d20 0x7f59240a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50318/0 (socket says 172.21.15.45:50318) 2024-09-16T03:22:04.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592259c640 1 -- 172.21.15.45:0/1514138297 learned_addr learned my addr 172.21.15.45:0/1514138297 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4049837839 0 0) 0x7f5924073360 con 0x7f59240a4d20 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient(hunting): my global_id is 144467 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5910003530 con 0x7f59240a4d20 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 528649697 0 0) 0x7f592409d9e0 con 0x7f59240a5810 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient(hunting): my global_id is 174144 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5924073360 con 0x7f59240a5810 2024-09-16T03:22:04.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1461694093 0 0) 0x7f5910003530 con 0x7f59240a4d20 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 >> v1:172.21.15.46:6789/0 conn(0x7f5924073d40 legacy=0x7f5924140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 >> v1:172.21.15.45:6789/0 conn(0x7f59240a5810 legacy=0x7f592413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: _finish_hunting 0 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 monclient: found mon.noname-b 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592409cf30 con 0x7f59240a4d20 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: _finish_auth 0 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: _check_auth_tickets 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f591c001a20 con 0x7f59240a4d20 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 5 monclient: authenticate success, global_id 144467 2024-09-16T03:22:04.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: dump: 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:04.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f591c001bc0 con 0x7f59240a4d20 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 1 -- 172.21.15.45:0/1514138297 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f591c002050 con 0x7f59240a4d20 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 4 set_mon_vals no callback set 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f592159a640 10 monclient: dump: 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.695 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: shutdown 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 -- 172.21.15.45:0/1514138297 >> v1:172.21.15.45:6790/0 conn(0x7f59240a4d20 legacy=0x7f59240a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 -- 172.21.15.45:0/1514138297 shutdown_connections 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 -- 172.21.15.45:0/1514138297 >> 172.21.15.45:0/1514138297 conn(0x7f59240aa720 msgr2=0x7f59240aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 -- 172.21.15.45:0/1514138297 shutdown_connections 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 1 -- 172.21.15.45:0/1514138297 wait complete. 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: build_initial_monmap 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.680+0000 7f5928c77640 10 monclient: monmap: 2024-09-16T03:22:04.696 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 librados: starting msgr at 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 librados: starting objecter 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 Processor -- start 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 -- start start 2024-09-16T03:22:04.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 librados: setting wanted keys 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 librados: calling monclient init 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient: init 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient(hunting): picked mon.noname-c con 0x7f5924073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient(hunting): picked mon.noname-a con 0x7f59240a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient(hunting): picked mon.noname-b con 0x7f59240a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59241405e0 con 0x7f59240a4d20 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59241408c0 con 0x7f59240a5810 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59240a1d00 con 0x7f5924073d40 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient(hunting): authenticate will time out at 2015.336843s 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f592259c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5924073d40 0x7f59241cd2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50320/0 (socket says 172.21.15.45:50320) 2024-09-16T03:22:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f592259c640 1 -- 172.21.15.45:0/421917604 learned_addr learned my addr 172.21.15.45:0/421917604 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.?.objecter ms_handle_connect 0x7f5924073d40 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.?.objecter ms_handle_connect 0x7f59240a4d20 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 907417373 0 0) 0x7f59241405e0 con 0x7f59240a4d20 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): my global_id is 174147 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5904004ec0 con 0x7f59240a4d20 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360480950 0 0) 0x7f59240a1d00 con 0x7f5924073d40 2024-09-16T03:22:04.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): my global_id is 144470 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59241405e0 con 0x7f5924073d40 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.?.objecter ms_handle_connect 0x7f59240a5810 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 329595959 0 0) 0x7f5904004ec0 con 0x7f59240a4d20 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59240a1d00 con 0x7f59240a4d20 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f590c0013c0 con 0x7f59240a4d20 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): dump: 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.700 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2662935320 0 0) 0x7f59241405e0 con 0x7f5924073d40 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5904004ec0 con 0x7f5924073d40 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f591c001f10 con 0x7f5924073d40 2024-09-16T03:22:04.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): dump: 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2044937925 0 0) 0x7f59241408c0 con 0x7f59240a5810 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): my global_id is 164245 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59241405e0 con 0x7f59240a5810 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1673313177 0 0) 0x7f59240a1d00 con 0x7f59240a4d20 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 >> v1:172.21.15.45:6790/0 conn(0x7f5924073d40 legacy=0x7f59241cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 >> v1:172.21.15.46:6789/0 conn(0x7f59240a5810 legacy=0x7f59241d5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: _finish_hunting 0 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 monclient: found mon.a 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59240a1510 con 0x7f59240a4d20 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: _finish_auth 0 2024-09-16T03:22:04.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: _check_auth_tickets 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 5 monclient: authenticate success, global_id 174147 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient: _renew_subs 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5924140300 con 0x7f59240a4d20 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 client.174147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient: _renew_subs 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f592413dd50 con 0x7f59240a4d20 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 librados: init done 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f590c003600 con 0x7f59240a4d20 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:04.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f590c003950 con 0x7f59240a4d20 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5920d99640 4 set_mon_vals no callback set 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: dump: 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.705 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f590c00f9c0 con 0x7f59240a4d20 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59080065d0 con 0x7f5908002870 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(696..696 src has 99..696) ==== 5917+0+0 (unknown 4201109949 0 0) 0x7f590c04a7d0 con 0x7f59240a4d20 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.174147.objecter ms_dispatch 0x7f5924000ba0 osd_map(696..696 src has 99..696) 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 3 client.174147.objecter handle_osd_map got epochs [696,696] > 0 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 3 client.174147.objecter handle_osd_map decoding full epoch 696 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 20 client.174147.objecter dump_active .. 0 homeless 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.174147.objecter ms_handle_connect 0x7f5908002870 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.174147.objecter resend_mon_ops 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4141923855 0 0) 0x7f59080065d0 con 0x7f5908002870 2024-09-16T03:22:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient(hunting): my global_id is 144473 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59240a1d00 con 0x7f5908002870 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f5922d9d640 10 monclient: get_auth_request con 0x7f590403e2b0 auth_method 0 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 898946362 0 0) 0x7f59240a1d00 con 0x7f5908002870 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59080065d0 con 0x7f5908002870 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f591c004cd0 con 0x7f5908002870 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 monclient: dump: 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3171866476 0 0) 0x7f59080065d0 con 0x7f5908002870 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f590800a170 con 0x7f5908002870 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.684+0000 7f591affd640 10 client.174147.objecter ms_handle_connect 0x7f590403e2b0 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.688+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f591c004ef0 con 0x7f5908002870 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.688+0000 7f591affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:04.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.688+0000 7f591affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:04.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.688+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 >> v1:172.21.15.45:6790/0 conn(0x7f5908002870 legacy=0x7f5908004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.688+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 >> v1:172.21.15.45:6790/0 conn(0x7f5908002870 legacy=0x7f5908004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:04.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:22:04.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5908009e30 con 0x7f59080090f0 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 client.174147.objecter ms_handle_connect 0x7f59080090f0 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 client.174147.objecter resend_mon_ops 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240205074 0 0) 0x7f5908009e30 con 0x7f59080090f0 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 monclient(hunting): my global_id is 144476 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59080065d0 con 0x7f59080090f0 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2650821178 0 0) 0x7f59080065d0 con 0x7f59080090f0 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5908009e30 con 0x7f59080090f0 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f591c0054b0 con 0x7f59080090f0 2024-09-16T03:22:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 monclient: dump: 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.712 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3713316772 0 0) 0x7f5908009e30 con 0x7f59080090f0 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f590800c400 con 0x7f59080090f0 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 2008904195) 0x7f591c007730 con 0x7f59080090f0 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 >> v1:172.21.15.45:6790/0 conn(0x7f59080090f0 legacy=0x7f5908009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f591affd640 1 -- 172.21.15.45:0/421917604 >> v1:172.21.15.45:6790/0 conn(0x7f59080090f0 legacy=0x7f5908009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:04.713 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 70, 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 349766, 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:22:04.714 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.715 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:22:04.717 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:04.718 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.719 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:04.720 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:22:04.720 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:04.720 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.720 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.720 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:22:04.720 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:22:04.720 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:22:04.721 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:04.722 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:04.723 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.724 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.726 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:22:04.726 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:22:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 10 librados: watch_flush enter 2024-09-16T03:22:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 10 librados: watch_flush exit 2024-09-16T03:22:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 20 client.174147.objecter shutdown clearing up homeless session... 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 10 client.174147.objecter successfully canceled tick 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 >> v1:172.21.15.46:6812/149360280 conn(0x7f590403e2b0 legacy=0x7f5904040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 10 monclient: shutdown 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 >> v1:172.21.15.45:6789/0 conn(0x7f59240a4d20 legacy=0x7f59241d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5922d9d640 1 -- 172.21.15.45:0/421917604 reap_dead start 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5922d9d640 1 -- 172.21.15.45:0/421917604 reap_dead start 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 shutdown_connections 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 >> 172.21.15.45:0/421917604 conn(0x7f59240aa720 msgr2=0x7f59240a40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.692+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 shutdown_connections 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.696+0000 7f5928c77640 1 -- 172.21.15.45:0/421917604 wait complete. 2024-09-16T03:22:04.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.696+0000 7f5928c77640 1 librados: shutdown 2024-09-16T03:22:04.729 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:22:04.729 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:22:04.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:22:04.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.840+0000 7f7bed1e8640 10 monclient: get_monmap_and_config 2024-09-16T03:22:04.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.840+0000 7f7bed1e8640 10 monclient: build_initial_monmap 2024-09-16T03:22:04.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.840+0000 7f7bed1e8640 10 monclient: monmap: 2024-09-16T03:22:04.845 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:04.845 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.846 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:04.847458+0000 2024-09-16T03:22:04.846 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:04.847458+0000 2024-09-16T03:22:04.846 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:04.846 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.846 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:04.846 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:04.846 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:04.846 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 Processor -- start 2024-09-16T03:22:04.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- start start 2024-09-16T03:22:04.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: init 2024-09-16T03:22:04.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:04.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:22:04.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f7be8137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:04.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): picked mon.noname-b con 0x7f7be8138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:04.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f7be813e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be80a4f90 con 0x7f7be813e2a0 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be80a4cb0 con 0x7f7be8137d20 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be809dc70 con 0x7f7be8138100 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): authenticate will time out at 2015.496836s 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f7be8138100 0x7f7be813cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50342/0 (socket says 172.21.15.45:50342) 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be659c640 1 -- 172.21.15.45:0/2153677263 learned_addr learned my addr 172.21.15.45:0/2153677263 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3902554630 0 0) 0x7f7be809dc70 con 0x7f7be8138100 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): my global_id is 144479 2024-09-16T03:22:04.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bd0003530 con 0x7f7be8138100 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 471177515 0 0) 0x7f7be80a4f90 con 0x7f7be813e2a0 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): my global_id is 174150 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7be809dc70 con 0x7f7be813e2a0 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353932457 0 0) 0x7f7be80a4cb0 con 0x7f7be8137d20 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient(hunting): my global_id is 164248 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7be80a4f90 con 0x7f7be8137d20 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2035343661 0 0) 0x7f7be809dc70 con 0x7f7be813e2a0 2024-09-16T03:22:04.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 >> v1:172.21.15.45:6790/0 conn(0x7f7be8138100 legacy=0x7f7be813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 >> v1:172.21.15.46:6789/0 conn(0x7f7be8137d20 legacy=0x7f7be80a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 monclient: found mon.noname-a 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be809cc50 con 0x7f7be813e2a0 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: _finish_auth 0 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7be0001a00 con 0x7f7be813e2a0 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: dump: 2024-09-16T03:22:04.852 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 5 monclient: authenticate success, global_id 174150 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7be0001ba0 con 0x7f7be813e2a0 2024-09-16T03:22:04.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 1 -- 172.21.15.45:0/2153677263 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7be0002030 con 0x7f7be813e2a0 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7be5d9b640 10 monclient: dump: 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.854 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 4 set_mon_vals no callback set 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: shutdown 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- 172.21.15.45:0/2153677263 >> v1:172.21.15.45:6789/0 conn(0x7f7be813e2a0 legacy=0x7f7be8140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- 172.21.15.45:0/2153677263 shutdown_connections 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- 172.21.15.45:0/2153677263 >> 172.21.15.45:0/2153677263 conn(0x7f7be80aa110 msgr2=0x7f7be80aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- 172.21.15.45:0/2153677263 shutdown_connections 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- 172.21.15.45:0/2153677263 wait complete. 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: build_initial_monmap 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: monmap: 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:04.855 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 librados: starting msgr at 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 librados: starting objecter 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 Processor -- start 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- start start 2024-09-16T03:22:04.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 librados: setting wanted keys 2024-09-16T03:22:04.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 librados: calling monclient init 2024-09-16T03:22:04.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: init 2024-09-16T03:22:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:22:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f7be8137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f7be8138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): picked mon.noname-b con 0x7f7be813e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be80a1300 con 0x7f7be8137d20 2024-09-16T03:22:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be813c050 con 0x7f7be813e2a0 2024-09-16T03:22:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be8139770 con 0x7f7be8138100 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.844+0000 7f7bed1e8640 10 monclient(hunting): authenticate will time out at 2015.499976s 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7be6d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7be8137d20 0x7f7be8138950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44734/0 (socket says 172.21.15.45:44734) 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7be6d9d640 1 -- 172.21.15.45:0/1507580058 learned_addr learned my addr 172.21.15.45:0/1507580058 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7be659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f7be8138100 0x7f7be8139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50346/0 (socket says 172.21.15.45:50346) 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.?.objecter ms_handle_connect 0x7f7be8137d20 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.?.objecter ms_handle_connect 0x7f7be8138100 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3829400551 0 0) 0x7f7be8139770 con 0x7f7be8138100 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): my global_id is 144482 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bd4004ec0 con 0x7f7be8138100 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3709571299 0 0) 0x7f7be80a4f90 con 0x7f7be8137d20 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): my global_id is 174153 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7be8139770 con 0x7f7be8137d20 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.?.objecter ms_handle_connect 0x7f7be813e2a0 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1464841041 0 0) 0x7f7be8139770 con 0x7f7be8137d20 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7be80a4f90 con 0x7f7be8137d20 2024-09-16T03:22:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 176936700 0 0) 0x7f7bd4004ec0 con 0x7f7be8138100 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7be8139770 con 0x7f7be8138100 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823756660 0 0) 0x7f7be813c050 con 0x7f7be813e2a0 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): my global_id is 164251 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bd4004ec0 con 0x7f7be813e2a0 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7bd8001960 con 0x7f7be8137d20 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): dump: 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.865 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7bdc001160 con 0x7f7be8138100 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): dump: 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3006070244 0 0) 0x7f7be80a4f90 con 0x7f7be8137d20 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 >> v1:172.21.15.45:6790/0 conn(0x7f7be8138100 legacy=0x7f7be8139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 >> v1:172.21.15.46:6789/0 conn(0x7f7be813e2a0 legacy=0x7f7be81d4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 monclient: found mon.a 2024-09-16T03:22:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be813dc30 con 0x7f7be8137d20 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: _finish_auth 0 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 5 monclient: authenticate success, global_id 174153 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 10 monclient: _renew_subs 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7be8139cf0 con 0x7f7be8137d20 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 10 client.174153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 10 monclient: _renew_subs 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:04.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7be813c470 con 0x7f7be8137d20 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 1 librados: init done 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7bd8001ca0 con 0x7f7be8137d20 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7bd8003a70 con 0x7f7be8137d20 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7be559a640 4 set_mon_vals no callback set 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: dump: 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.869 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7bc80065d0 con 0x7f7bc8002870 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f7bd8003c10 con 0x7f7be8137d20 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(696..696 src has 99..696) ==== 5917+0+0 (unknown 4201109949 0 0) 0x7f7bd804b920 con 0x7f7be8137d20 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.174153.objecter ms_dispatch 0x7f7be8000ba0 osd_map(696..696 src has 99..696) 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 3 client.174153.objecter handle_osd_map got epochs [696,696] > 0 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 3 client.174153.objecter handle_osd_map decoding full epoch 696 2024-09-16T03:22:04.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 20 client.174153.objecter dump_active .. 0 homeless 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.174153.objecter ms_handle_connect 0x7f7bc8002870 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.174153.objecter resend_mon_ops 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7be759e640 10 monclient: get_auth_request con 0x7f7bd403e4b0 auth_method 0 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2356688033 0 0) 0x7f7bc80065d0 con 0x7f7bc8002870 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient(hunting): my global_id is 164254 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7be80a4f90 con 0x7f7bc8002870 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 client.174153.objecter ms_handle_connect 0x7f7bd403e4b0 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 194118706 0 0) 0x7f7be80a4f90 con 0x7f7bc8002870 2024-09-16T03:22:04.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7bc80065d0 con 0x7f7bc8002870 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7bdc004b80 con 0x7f7bc8002870 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 10 monclient: dump: 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.872 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1583488967 0 0) 0x7f7bc80065d0 con 0x7f7bc8002870 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.848+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7bc800a170 con 0x7f7bc8002870 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.852+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7bdc006160 con 0x7f7bc8002870 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.852+0000 7f7bcf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.852+0000 7f7bcf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.852+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 >> v1:172.21.15.46:6789/0 conn(0x7f7bc8002870 legacy=0x7f7bc8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.852+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 >> v1:172.21.15.46:6789/0 conn(0x7f7bc8002870 legacy=0x7f7bc8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bed1e8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7bc8009e10 con 0x7f7bc80090f0 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 10 client.174153.objecter ms_handle_connect 0x7f7bc80090f0 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 10 client.174153.objecter resend_mon_ops 2024-09-16T03:22:04.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2760250887 0 0) 0x7f7bc8009e10 con 0x7f7bc80090f0 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 10 monclient(hunting): my global_id is 164257 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bc80065d0 con 0x7f7bc80090f0 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4227135159 0 0) 0x7f7bc80065d0 con 0x7f7bc80090f0 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7bc8009e10 con 0x7f7bc80090f0 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f7bdc006830 con 0x7f7bc80090f0 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 10 monclient: dump: 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 953297353 0 0) 0x7f7bc8009e10 con 0x7f7bc80090f0 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.856+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7bc800c3b0 con 0x7f7bc80090f0 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 1314006878) 0x7f7bdc006650 con 0x7f7bc80090f0 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bcf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bcf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 >> v1:172.21.15.46:6789/0 conn(0x7f7bc80090f0 legacy=0x7f7bc8009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bcf7fe640 1 -- 172.21.15.45:0/1507580058 >> v1:172.21.15.46:6789/0 conn(0x7f7bc80090f0 legacy=0x7f7bc8009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:04.876 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:22:04.876 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:04.876 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:04.876 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:22:04.876 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 70, 2024-09-16T03:22:04.876 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 127170, 2024-09-16T03:22:04.876 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:22:04.876 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 2, 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:04.877 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:04.878 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:22:04.879 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:04.880 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:22:04.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:22:04.882 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:04.883 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:04.884 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:04.885 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.886 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:04.887 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:04.888 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:04.888 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:04.888 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:04.888 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:04.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:04.888 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:22:04.888 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 10 librados: watch_flush enter 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 10 librados: watch_flush exit 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 20 client.174153.objecter shutdown clearing up homeless session... 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 10 client.174153.objecter successfully canceled tick 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 >> v1:172.21.15.46:6812/149360280 conn(0x7f7bd403e4b0 legacy=0x7f7bd4040970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 10 monclient: shutdown 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 >> v1:172.21.15.45:6789/0 conn(0x7f7be8137d20 legacy=0x7f7be8138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7be759e640 1 -- 172.21.15.45:0/1507580058 reap_dead start 2024-09-16T03:22:04.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7be759e640 1 -- 172.21.15.45:0/1507580058 reap_dead start 2024-09-16T03:22:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 shutdown_connections 2024-09-16T03:22:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 >> 172.21.15.45:0/1507580058 conn(0x7f7be80aa110 msgr2=0x7f7be80742a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 shutdown_connections 2024-09-16T03:22:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 1 -- 172.21.15.45:0/1507580058 wait complete. 2024-09-16T03:22:04.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:04.860+0000 7f7bed1e8640 1 librados: shutdown 2024-09-16T03:22:04.891 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:22:04.891 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:22:04.891 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:22:04.892 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:22:04.892 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:22:04.892 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:22:04.892 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:22:04.906 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:22:04.906 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:22:04.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:22:05.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient: get_monmap_and_config 2024-09-16T03:22:05.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient: build_initial_monmap 2024-09-16T03:22:05.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient: monmap: 2024-09-16T03:22:05.072 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:05.072 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:05.072 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:05.074316+0000 2024-09-16T03:22:05.073 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:05.074316+0000 2024-09-16T03:22:05.073 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:05.073 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:05.073 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:05.073 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:05.073 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:05.074 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 1 Processor -- start 2024-09-16T03:22:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 1 -- start start 2024-09-16T03:22:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient: init 2024-09-16T03:22:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:22:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient(hunting): picked mon.noname-a con 0x7f5524135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient(hunting): picked mon.noname-c con 0x7f552409fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient(hunting): picked mon.noname-b con 0x7f552413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f552409f7e0 con 0x7f5524135950 2024-09-16T03:22:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f552409b160 con 0x7f552409fc40 2024-09-16T03:22:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f552409b440 con 0x7f552413c010 2024-09-16T03:22:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f552b4aa640 10 monclient(hunting): authenticate will time out at 2015.723619s 2024-09-16T03:22:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f5529a47640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f552413c010 legacy=0x7f552413e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f5529246640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5524135950 0x7f5524135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44768/0 (socket says 172.21.15.45:44768) 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.068+0000 7f5529246640 1 -- 172.21.15.45:0/1973998211 learned_addr learned my addr 172.21.15.45:0/1973998211 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 -- 172.21.15.45:0/1973998211 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4164449785 0 0) 0x7f552409f7e0 con 0x7f5524135950 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient(hunting): _negotiate 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient(hunting): my global_id is 174156 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 -- 172.21.15.45:0/1973998211 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5510003530 con 0x7f5524135950 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 -- 172.21.15.45:0/1973998211 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2443658507 0 0) 0x7f5510003530 con 0x7f5524135950 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 -- 172.21.15.45:0/1973998211 >> v1:172.21.15.45:6790/0 conn(0x7f552413c010 legacy=0x7f552413e400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:22:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 -- 172.21.15.45:0/1973998211 >> v1:172.21.15.46:6789/0 conn(0x7f552409fc40 legacy=0x7f552413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: _finish_hunting 0 2024-09-16T03:22:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 monclient: found mon.noname-a 2024-09-16T03:22:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:22:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 -- 172.21.15.45:0/1973998211 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5524136440 con 0x7f5524135950 2024-09-16T03:22:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: _finish_auth 0 2024-09-16T03:22:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: _check_auth_tickets 2024-09-16T03:22:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 5 monclient: authenticate success, global_id 174156 2024-09-16T03:22:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 -- 172.21.15.45:0/1973998211 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f55180016a0 con 0x7f5524135950 2024-09-16T03:22:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: dump: 2024-09-16T03:22:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 -- 172.21.15.45:0/1973998211 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f55180018c0 con 0x7f5524135950 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 1 -- 172.21.15.45:0/1973998211 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5518001d50 con 0x7f5524135950 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 4 set_mon_vals no callback set 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5517fff640 10 monclient: dump: 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: shutdown 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- 172.21.15.45:0/1973998211 >> v1:172.21.15.45:6789/0 conn(0x7f5524135950 legacy=0x7f5524135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- 172.21.15.45:0/1973998211 shutdown_connections 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- 172.21.15.45:0/1973998211 >> 172.21.15.45:0/1973998211 conn(0x7f55240a2bd0 msgr2=0x7f55240a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- 172.21.15.45:0/1973998211 shutdown_connections 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- 172.21.15.45:0/1973998211 wait complete. 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: build_initial_monmap 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: monmap: 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:05.088 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 librados: starting msgr at 2024-09-16T03:22:05.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 librados: starting objecter 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 Processor -- start 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- start start 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 librados: setting wanted keys 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 librados: calling monclient init 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: init 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient(hunting): picked mon.noname-b con 0x7f552409fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient(hunting): picked mon.noname-a con 0x7f5524135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient(hunting): picked mon.noname-c con 0x7f552413c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5524073990 con 0x7f5524135950 2024-09-16T03:22:05.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f552409d2c0 con 0x7f552409fc40 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55240735e0 con 0x7f552413c010 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient(hunting): authenticate will time out at 2015.726613s 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5529a47640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f552413c010 legacy=0x7f55241d4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5528a45640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5524135950 0x7f5524072ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44782/0 (socket says 172.21.15.45:44782) 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5528a45640 1 -- 172.21.15.45:0/3809917414 learned_addr learned my addr 172.21.15.45:0/3809917414 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 client.?.objecter ms_handle_connect 0x7f5524135950 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435432475 0 0) 0x7f5524073990 con 0x7f5524135950 2024-09-16T03:22:05.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:22:05.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:05.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:05.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient(hunting): my global_id is 174159 2024-09-16T03:22:05.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5508004ef0 con 0x7f5524135950 2024-09-16T03:22:05.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 client.?.objecter ms_handle_connect 0x7f552409fc40 2024-09-16T03:22:05.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:05.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3915325364 0 0) 0x7f5508004ef0 con 0x7f5524135950 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5524073990 con 0x7f5524135950 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5520001d20 con 0x7f5524135950 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient(hunting): dump: 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:05.093 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1171497594 0 0) 0x7f5524073990 con 0x7f5524135950 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 >> v1:172.21.15.45:6790/0 conn(0x7f552413c010 legacy=0x7f55241d4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 >> v1:172.21.15.46:6789/0 conn(0x7f552409fc40 legacy=0x7f55240727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 monclient: found mon.a 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552409ed90 con 0x7f5524135950 2024-09-16T03:22:05.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient: _finish_auth 0 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 5 monclient: authenticate success, global_id 174159 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: _renew_subs 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f552413bb90 con 0x7f5524135950 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 client.174159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: _renew_subs 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f552409eab0 con 0x7f5524135950 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f552b4aa640 1 librados: init done 2024-09-16T03:22:05.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5520002060 con 0x7f5524135950 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f5520003df0 con 0x7f5524135950 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f5515ffb640 10 monclient: dump: 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:05.096 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.072+0000 7f55177fe640 4 set_mon_vals no callback set 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f552b4aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f552b4aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f552b4aa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f5520004010 con 0x7f5524135950 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f552406c3e0 con 0x7f5524135950 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(696..696 src has 99..696) ==== 5917+0+0 (unknown 4201109949 0 0) 0x7f552004be10 con 0x7f5524135950 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f5515ffb640 10 client.174159.objecter ms_dispatch 0x7f5524000ba0 osd_map(696..696 src has 99..696) 2024-09-16T03:22:05.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f5515ffb640 3 client.174159.objecter handle_osd_map got epochs [696,696] > 0 2024-09-16T03:22:05.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f5515ffb640 3 client.174159.objecter handle_osd_map decoding full epoch 696 2024-09-16T03:22:05.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f5515ffb640 20 client.174159.objecter dump_active .. 0 homeless 2024-09-16T03:22:05.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f5529246640 10 monclient: get_auth_request con 0x7f550803df50 auth_method 0 2024-09-16T03:22:05.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.076+0000 7f5515ffb640 10 client.174159.objecter ms_handle_connect 0x7f550803df50 2024-09-16T03:22:05.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.080+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5520016e80 con 0x7f5524135950 2024-09-16T03:22:05.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.080+0000 7f5515ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:05.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.080+0000 7f5515ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:05.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:22:05.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:05.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:05.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f55241355b0 con 0x7f5524135950 2024-09-16T03:22:05.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f5515ffb640 1 -- 172.21.15.45:0/3809917414 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2579252555) 0x7f5520016780 con 0x7f5524135950 2024-09-16T03:22:05.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f5515ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:05.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f5515ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:05.324 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:22:05.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 10 librados: watch_flush enter 2024-09-16T03:22:05.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 10 librados: watch_flush exit 2024-09-16T03:22:05.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 20 client.174159.objecter shutdown clearing up homeless session... 2024-09-16T03:22:05.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 10 client.174159.objecter successfully canceled tick 2024-09-16T03:22:05.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 >> v1:172.21.15.46:6812/149360280 conn(0x7f550803df50 legacy=0x7f5508040410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:05.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 10 monclient: shutdown 2024-09-16T03:22:05.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:05.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.320+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 >> v1:172.21.15.45:6789/0 conn(0x7f5524135950 legacy=0x7f5524072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:05.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.324+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 shutdown_connections 2024-09-16T03:22:05.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.324+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 >> 172.21.15.45:0/3809917414 conn(0x7f55240a2bd0 msgr2=0x7f552413eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:05.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.324+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 shutdown_connections 2024-09-16T03:22:05.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.324+0000 7f552b4aa640 1 -- 172.21.15.45:0/3809917414 wait complete. 2024-09-16T03:22:05.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:05.324+0000 7f552b4aa640 1 librados: shutdown 2024-09-16T03:22:08.345 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:22:08.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient: get_monmap_and_config 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient: build_initial_monmap 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient: monmap: 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:08.481850+0000 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:08.481850+0000 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:08.481 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 1 Processor -- start 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 1 -- start start 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient: init 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient(hunting): picked mon.noname-a con 0x7f119c0a25d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient(hunting): picked mon.noname-b con 0x7f119c09a850 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient(hunting): picked mon.noname-c con 0x7f119c09ac60 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119c09c3f0 con 0x7f119c0a25d0 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119c09c6d0 con 0x7f119c09ac60 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119c09c9b0 con 0x7f119c09a850 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f11a2339640 10 monclient(hunting): authenticate will time out at 2019.131936s 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.476+0000 7f1193fff640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f119c09a850 legacy=0x7f119c139250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f119c0a25d0 0x7f119c09a140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44792/0 (socket says 172.21.15.45:44792) 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119bfff640 1 -- 172.21.15.45:0/2655450177 learned_addr learned my addr 172.21.15.45:0/2655450177 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 -- 172.21.15.45:0/2655450177 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3700271398 0 0) 0x7f119c09c3f0 con 0x7f119c0a25d0 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient(hunting): my global_id is 174162 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 -- 172.21.15.45:0/2655450177 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1188004640 con 0x7f119c0a25d0 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 -- 172.21.15.45:0/2655450177 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1271813308 0 0) 0x7f1188004640 con 0x7f119c0a25d0 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 -- 172.21.15.45:0/2655450177 >> v1:172.21.15.45:6790/0 conn(0x7f119c09a850 legacy=0x7f119c139250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:22:08.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 -- 172.21.15.45:0/2655450177 >> v1:172.21.15.46:6789/0 conn(0x7f119c09ac60 legacy=0x7f119c13c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:22:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:22:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:22:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 -- 172.21.15.45:0/2655450177 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c09b350 con 0x7f119c0a25d0 2024-09-16T03:22:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:22:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:22:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 -- 172.21.15.45:0/2655450177 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f118c0016a0 con 0x7f119c0a25d0 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: dump: 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 5 monclient: authenticate success, global_id 174162 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 -- 172.21.15.45:0/2655450177 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f118c0018c0 con 0x7f119c0a25d0 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 1 -- 172.21.15.45:0/2655450177 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f118c001cd0 con 0x7f119c0a25d0 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f119b7fe640 10 monclient: dump: 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 4 set_mon_vals no callback set 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient: shutdown 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- 172.21.15.45:0/2655450177 >> v1:172.21.15.45:6789/0 conn(0x7f119c0a25d0 legacy=0x7f119c09a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- 172.21.15.45:0/2655450177 shutdown_connections 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- 172.21.15.45:0/2655450177 >> 172.21.15.45:0/2655450177 conn(0x7f119c0a2bd0 msgr2=0x7f119c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- 172.21.15.45:0/2655450177 shutdown_connections 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- 172.21.15.45:0/2655450177 wait complete. 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient: build_initial_monmap 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient: monmap: 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:08.490 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:08.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 librados: starting msgr at 2024-09-16T03:22:08.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 librados: starting objecter 2024-09-16T03:22:08.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 Processor -- start 2024-09-16T03:22:08.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- start start 2024-09-16T03:22:08.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 librados: setting wanted keys 2024-09-16T03:22:08.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 librados: calling monclient init 2024-09-16T03:22:08.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient: init 2024-09-16T03:22:08.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:08.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:22:08.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient(hunting): picked mon.noname-b con 0x7f119c09a850 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:08.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient(hunting): picked mon.noname-c con 0x7f119c09ac60 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:08.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient(hunting): picked mon.noname-a con 0x7f119c0a25d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:08.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119c097ba0 con 0x7f119c0a25d0 2024-09-16T03:22:08.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119c09d680 con 0x7f119c09a850 2024-09-16T03:22:08.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119c097e80 con 0x7f119c09ac60 2024-09-16T03:22:08.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:08.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient(hunting): authenticate will time out at 2019.135016s 2024-09-16T03:22:08.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f1193fff640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f119c09ac60 legacy=0x7f119c1d1910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:08.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a08d6640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f119c0a25d0 0x7f119c1d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44798/0 (socket says 172.21.15.45:44798) 2024-09-16T03:22:08.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a08d6640 1 -- 172.21.15.45:0/4265908629 learned_addr learned my addr 172.21.15.45:0/4265908629 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:08.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 client.?.objecter ms_handle_connect 0x7f119c0a25d0 2024-09-16T03:22:08.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:08.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1701041721 0 0) 0x7f119c097ba0 con 0x7f119c0a25d0 2024-09-16T03:22:08.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient(hunting): _negotiate 2024-09-16T03:22:08.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:08.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:08.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient(hunting): my global_id is 174165 2024-09-16T03:22:08.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f117c004ec0 con 0x7f119c0a25d0 2024-09-16T03:22:08.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 client.?.objecter ms_handle_connect 0x7f119c09a850 2024-09-16T03:22:08.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:08.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3224075191 0 0) 0x7f117c004ec0 con 0x7f119c0a25d0 2024-09-16T03:22:08.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f119c097ba0 con 0x7f119c0a25d0 2024-09-16T03:22:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1194001d40 con 0x7f119c0a25d0 2024-09-16T03:22:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient(hunting): dump: 2024-09-16T03:22:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:08.500 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:08.501 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1704212167 0 0) 0x7f119c097ba0 con 0x7f119c0a25d0 2024-09-16T03:22:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 >> v1:172.21.15.45:6790/0 conn(0x7f119c09ac60 legacy=0x7f119c1d1910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:22:08.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 >> v1:172.21.15.46:6789/0 conn(0x7f119c09a850 legacy=0x7f119c06e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient: _finish_hunting 0 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 monclient: found mon.a 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c09a4a0 con 0x7f119c0a25d0 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient: _finish_auth 0 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 10 monclient: _check_auth_tickets 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 5 monclient: authenticate success, global_id 174165 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient: _renew_subs 2024-09-16T03:22:08.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:08.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.480+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f119c09d2c0 con 0x7f119c0a25d0 2024-09-16T03:22:08.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11a2339640 10 client.174165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:08.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11a2339640 10 monclient: _renew_subs 2024-09-16T03:22:08.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11a2339640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:08.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f119c099e20 con 0x7f119c0a25d0 2024-09-16T03:22:08.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11a2339640 1 librados: init done 2024-09-16T03:22:08.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1194002000 con 0x7f119c0a25d0 2024-09-16T03:22:08.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:08.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1194003df0 con 0x7f119c0a25d0 2024-09-16T03:22:08.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:08.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:08.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f119affd640 4 set_mon_vals no callback set 2024-09-16T03:22:08.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 10 monclient: dump: 2024-09-16T03:22:08.504 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:08.504 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:08.504 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:08.505 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:08.505 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:08.505 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:08.505 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:08.505 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:08.505 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:08.505 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:08.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11a2339640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:08.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11a2339640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:08.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11a2339640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:08.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1180005090 con 0x7f119c0a25d0 2024-09-16T03:22:08.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f1194003f90 con 0x7f119c0a25d0 2024-09-16T03:22:08.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(696..696 src has 99..696) ==== 5917+0+0 (unknown 4201109949 0 0) 0x7f119404bcf0 con 0x7f119c0a25d0 2024-09-16T03:22:08.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 10 client.174165.objecter ms_dispatch 0x7f119c000ba0 osd_map(696..696 src has 99..696) 2024-09-16T03:22:08.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 3 client.174165.objecter handle_osd_map got epochs [696,696] > 0 2024-09-16T03:22:08.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 3 client.174165.objecter handle_osd_map decoding full epoch 696 2024-09-16T03:22:08.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 20 client.174165.objecter dump_active .. 0 homeless 2024-09-16T03:22:08.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f119bfff640 10 monclient: get_auth_request con 0x7f117c03df70 auth_method 0 2024-09-16T03:22:08.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.484+0000 7f11997fa640 10 client.174165.objecter ms_handle_connect 0x7f117c03df70 2024-09-16T03:22:08.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.488+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1194016770 con 0x7f119c0a25d0 2024-09-16T03:22:08.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.488+0000 7f11997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:08.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.488+0000 7f11997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:08.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.728+0000 7f11a2339640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:22:08.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.728+0000 7f11a2339640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:08.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.728+0000 7f11a2339640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:08.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.728+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1180005380 con 0x7f119c0a25d0 2024-09-16T03:22:08.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.728+0000 7f11997fa640 1 -- 172.21.15.45:0/4265908629 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 2797531211) 0x7f1194020020 con 0x7f119c0a25d0 2024-09-16T03:22:08.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.728+0000 7f11997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:08.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.728+0000 7f11997fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:08.734 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:22:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 10 librados: watch_flush enter 2024-09-16T03:22:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 10 librados: watch_flush exit 2024-09-16T03:22:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 20 client.174165.objecter shutdown clearing up homeless session... 2024-09-16T03:22:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 10 client.174165.objecter successfully canceled tick 2024-09-16T03:22:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 >> v1:172.21.15.46:6812/149360280 conn(0x7f117c03df70 legacy=0x7f117c040430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 10 monclient: shutdown 2024-09-16T03:22:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 >> v1:172.21.15.45:6789/0 conn(0x7f119c0a25d0 legacy=0x7f119c1d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 shutdown_connections 2024-09-16T03:22:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 >> 172.21.15.45:0/4265908629 conn(0x7f119c0a2bd0 msgr2=0x7f119c0a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 shutdown_connections 2024-09-16T03:22:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 1 -- 172.21.15.45:0/4265908629 wait complete. 2024-09-16T03:22:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:08.732+0000 7f11a2339640 1 librados: shutdown 2024-09-16T03:22:11.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:22:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.880+0000 7fd66054d640 10 monclient: get_monmap_and_config 2024-09-16T03:22:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.880+0000 7fd66054d640 10 monclient: build_initial_monmap 2024-09-16T03:22:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.880+0000 7fd66054d640 10 monclient: monmap: 2024-09-16T03:22:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:11.884 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:11.886376+0000 2024-09-16T03:22:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:11.886376+0000 2024-09-16T03:22:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:11.885 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:11.885 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:11.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.880+0000 7fd66054d640 1 Processor -- start 2024-09-16T03:22:11.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.880+0000 7fd66054d640 1 -- start start 2024-09-16T03:22:11.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.880+0000 7fd66054d640 10 monclient: init 2024-09-16T03:22:11.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:11.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:22:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 10 monclient(hunting): picked mon.noname-c con 0x7fd65809b8e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 10 monclient(hunting): picked mon.noname-a con 0x7fd65809b250 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 10 monclient(hunting): picked mon.noname-b con 0x7fd65813b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65809cc20 con 0x7fd65809b250 2024-09-16T03:22:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65809d820 con 0x7fd65809b8e0 2024-09-16T03:22:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65809b6a0 con 0x7fd65813b000 2024-09-16T03:22:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd66054d640 10 monclient(hunting): authenticate will time out at 2022.536266s 2024-09-16T03:22:11.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd65eaea640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd65813d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:11.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd65dae8640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd65809b250 0x7fd6581398e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58480/0 (socket says 172.21.15.45:58480) 2024-09-16T03:22:11.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:11.884+0000 7fd65dae8640 1 -- 172.21.15.45:0/2258294768 learned_addr learned my addr 172.21.15.45:0/2258294768 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:12.088+0000 7fd65eaea640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd65813d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:12.492+0000 7fd65eaea640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd65813d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:13.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:13.296+0000 7fd65eaea640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd65813d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 10 monclient(hunting): tick 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 1 monclient(hunting): continuing hunt 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd65813d3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.46:6789/0 conn(0x7fd65809b8e0 legacy=0x7fd65809bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6789/0 conn(0x7fd65809b250 legacy=0x7fd6581398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fd63c003700 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fd65813b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fd63c00ab90 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6580a21f0 con 0x7fd63c003700 2024-09-16T03:22:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65809b6a0 con 0x7fd63c00ab90 2024-09-16T03:22:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c010920 con 0x7fd65813b000 2024-09-16T03:22:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd64ffff640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:14.884+0000 7fd65dae8640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd63c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:15.088+0000 7fd65dae8640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd63c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:15.492+0000 7fd65dae8640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd63c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:16.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:16.296+0000 7fd65dae8640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd63c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:17.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 10 monclient(hunting): tick 2024-09-16T03:22:17.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:22:17.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 1 monclient(hunting): continuing hunt 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd63c009420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.46:6789/0 conn(0x7fd63c00ab90 legacy=0x7fd63c00d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6789/0 conn(0x7fd63c003700 legacy=0x7fd63c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fd63c010c20 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fd63c00ab90 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fd65813b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c011c00 con 0x7fd63c010c20 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c010920 con 0x7fd63c00ab90 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 1 -- 172.21.15.45:0/2258294768 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c011900 con 0x7fd65813b000 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd64ffff640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:17.884+0000 7fd65eaea640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd63c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:18.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:18.088+0000 7fd65eaea640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd63c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:18.492+0000 7fd65eaea640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd63c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:19.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.076+0000 7fd65d2e7640 1 -- 172.21.15.45:0/2258294768 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 594765614 0 0) 0x7fd63c011c00 con 0x7fd63c010c20 2024-09-16T03:22:19.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.076+0000 7fd65d2e7640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.076+0000 7fd65d2e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.076+0000 7fd65d2e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.076+0000 7fd65d2e7640 10 monclient(hunting): my global_id is 174168 2024-09-16T03:22:19.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.076+0000 7fd65d2e7640 1 -- 172.21.15.45:0/2258294768 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd644005740 con 0x7fd63c010c20 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 1 -- 172.21.15.45:0/2258294768 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1945410625 0 0) 0x7fd644005740 con 0x7fd63c010c20 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6790/0 conn(0x7fd65813b000 legacy=0x7fd63c009420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.46:6789/0 conn(0x7fd63c00ab90 legacy=0x7fd63c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: _finish_hunting 0 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 1 monclient: found mon.noname-a 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 1 -- 172.21.15.45:0/2258294768 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c00d0d0 con 0x7fd63c010c20 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: _finish_auth 0 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: _check_auth_tickets 2024-09-16T03:22:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 1 -- 172.21.15.45:0/2258294768 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd654001560 con 0x7fd63c010c20 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd66054d640 5 monclient: authenticate success, global_id 174168 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd66054d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: dump: 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd66054d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 1 -- 172.21.15.45:0/2258294768 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd654001700 con 0x7fd63c010c20 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd66054d640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 1 -- 172.21.15.45:0/2258294768 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd654001b90 con 0x7fd63c010c20 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd66054d640 4 set_mon_vals no callback set 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.080+0000 7fd65d2e7640 10 monclient: dump: 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.088 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.088 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.088 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.088 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.088 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.088 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.088 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.088 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.084+0000 7fd66054d640 10 monclient: shutdown 2024-09-16T03:22:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.084+0000 7fd66054d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.084+0000 7fd66054d640 1 -- 172.21.15.45:0/2258294768 >> v1:172.21.15.45:6789/0 conn(0x7fd63c010c20 legacy=0x7fd63c011000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.084+0000 7fd66054d640 1 -- 172.21.15.45:0/2258294768 shutdown_connections 2024-09-16T03:22:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.084+0000 7fd66054d640 1 -- 172.21.15.45:0/2258294768 >> 172.21.15.45:0/2258294768 conn(0x7fd6580a2bd0 msgr2=0x7fd6580a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:19.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.092+0000 7fd66054d640 1 -- 172.21.15.45:0/2258294768 shutdown_connections 2024-09-16T03:22:19.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.092+0000 7fd66054d640 1 -- 172.21.15.45:0/2258294768 wait complete. 2024-09-16T03:22:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient: build_initial_monmap 2024-09-16T03:22:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient: monmap: 2024-09-16T03:22:19.102 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:19.103 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:19.103 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 1 librados: starting msgr at 2024-09-16T03:22:19.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 1 librados: starting objecter 2024-09-16T03:22:19.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 1 Processor -- start 2024-09-16T03:22:19.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 1 -- start start 2024-09-16T03:22:19.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 1 librados: setting wanted keys 2024-09-16T03:22:19.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 1 librados: calling monclient init 2024-09-16T03:22:19.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient: init 2024-09-16T03:22:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:22:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd65e2e9640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fd63c00ab90 legacy=0x7fd658091340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient(hunting): picked mon.noname-c con 0x7fd63c00ab90 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:19.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient(hunting): picked mon.noname-a con 0x7fd63c010c20 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient(hunting): picked mon.noname-b con 0x7fd65813b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd658092d80 con 0x7fd63c010c20 2024-09-16T03:22:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6580950e0 con 0x7fd65813b000 2024-09-16T03:22:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd658074100 con 0x7fd63c00ab90 2024-09-16T03:22:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd66054d640 10 monclient(hunting): authenticate will time out at 2029.755132s 2024-09-16T03:22:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd65dae8640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd63c010c20 0x7fd658091a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58500/0 (socket says 172.21.15.45:58500) 2024-09-16T03:22:19.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.100+0000 7fd65dae8640 1 -- 172.21.15.45:0/172539472 learned_addr learned my addr 172.21.15.45:0/172539472 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:19.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 client.?.objecter ms_handle_connect 0x7fd63c010c20 2024-09-16T03:22:19.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:19.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 client.?.objecter ms_handle_connect 0x7fd65813b000 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1506844404 0 0) 0x7fd658092d80 con 0x7fd63c010c20 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient(hunting): my global_id is 174171 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd63c011900 con 0x7fd63c010c20 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1638129076 0 0) 0x7fd63c011900 con 0x7fd63c010c20 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd658092d80 con 0x7fd63c010c20 2024-09-16T03:22:19.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd6480018c0 con 0x7fd63c010c20 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient(hunting): dump: 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2802704442 0 0) 0x7fd658092d80 con 0x7fd63c010c20 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 >> v1:172.21.15.45:6790/0 conn(0x7fd63c00ab90 legacy=0x7fd658091340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 >> v1:172.21.15.46:6789/0 conn(0x7fd65813b000 legacy=0x7fd658092160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient: _finish_hunting 0 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 monclient: found mon.a 2024-09-16T03:22:19.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65809d2c0 con 0x7fd63c010c20 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient: _finish_auth 0 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 10 monclient: _check_auth_tickets 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd64effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd66054d640 5 monclient: authenticate success, global_id 174171 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd66054d640 10 monclient: _renew_subs 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd66054d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6580a47d0 con 0x7fd63c010c20 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.104+0000 7fd66054d640 10 client.174171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:19.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd66054d640 10 monclient: _renew_subs 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd66054d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd6581d21e0 con 0x7fd63c010c20 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd66054d640 1 librados: init done 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6480047f0 con 0x7fd63c010c20 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd648004c80 con 0x7fd63c010c20 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 10 monclient: dump: 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.116 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.116 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.116 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.116 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.116 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.116 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd65cae6640 4 set_mon_vals no callback set 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd66054d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd66054d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd66054d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd640005090 con 0x7fd63c010c20 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fd648004e20 con 0x7fd63c010c20 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(697..697 src has 99..697) ==== 5930+0+0 (unknown 3213671683 0 0) 0x7fd64804b9c0 con 0x7fd63c010c20 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 10 client.174171.objecter ms_dispatch 0x7fd658000ba0 osd_map(697..697 src has 99..697) 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 3 client.174171.objecter handle_osd_map got epochs [697,697] > 0 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 3 client.174171.objecter handle_osd_map decoding full epoch 697 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 20 client.174171.objecter dump_active .. 0 homeless 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd65e2e9640 10 monclient: get_auth_request con 0x7fd63c03f360 auth_method 0 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.108+0000 7fd64effd640 10 client.174171.objecter ms_handle_connect 0x7fd63c03f360 2024-09-16T03:22:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.112+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd648016ae0 con 0x7fd63c010c20 2024-09-16T03:22:19.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.112+0000 7fd64effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:19.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.112+0000 7fd64effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:19.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:22:19.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:19.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd640005380 con 0x7fd63c010c20 2024-09-16T03:22:19.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd64effd640 1 -- 172.21.15.45:0/172539472 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 240907183) 0x7fd64804b7e0 con 0x7fd63c010c20 2024-09-16T03:22:19.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd64effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:22:19.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd64effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:19.355 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":74,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:22:19.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 10 librados: watch_flush enter 2024-09-16T03:22:19.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 10 librados: watch_flush exit 2024-09-16T03:22:19.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 20 client.174171.objecter shutdown clearing up homeless session... 2024-09-16T03:22:19.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 10 client.174171.objecter successfully canceled tick 2024-09-16T03:22:19.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 >> v1:172.21.15.46:6812/149360280 conn(0x7fd63c03f360 legacy=0x7fd63c041820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 10 monclient: shutdown 2024-09-16T03:22:19.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:19.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 >> v1:172.21.15.45:6789/0 conn(0x7fd63c010c20 legacy=0x7fd658091a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 shutdown_connections 2024-09-16T03:22:19.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 >> 172.21.15.45:0/172539472 conn(0x7fd6580a2bd0 msgr2=0x7fd6580707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:19.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 shutdown_connections 2024-09-16T03:22:19.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 1 -- 172.21.15.45:0/172539472 wait complete. 2024-09-16T03:22:19.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.352+0000 7fd66054d640 1 librados: shutdown 2024-09-16T03:22:19.374 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:22:19.374 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:22:19.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:22:19.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: get_monmap_and_config 2024-09-16T03:22:19.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: build_initial_monmap 2024-09-16T03:22:19.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: monmap: 2024-09-16T03:22:19.494 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:19.494 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.494 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:19.496453+0000 2024-09-16T03:22:19.494 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:19.496453+0000 2024-09-16T03:22:19.495 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:19.495 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.495 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:19.495 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:19.495 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:19.495 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 Processor -- start 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 -- start start 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: init 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient(hunting): picked mon.noname-a con 0x7fe05c0a4d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient(hunting): picked mon.noname-c con 0x7fe05c0a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient(hunting): picked mon.noname-b con 0x7fe05c073d40 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c09d9e0 con 0x7fe05c0a4d20 2024-09-16T03:22:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c09dc20 con 0x7fe05c0a5810 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c073360 con 0x7fe05c073d40 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient(hunting): authenticate will time out at 2030.145792s 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe062231640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fe05c073d40 legacy=0x7fe05c140660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe061a30640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe05c0a4d20 0x7fe05c0a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58514/0 (socket says 172.21.15.45:58514) 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe061a30640 1 -- 172.21.15.45:0/2513981754 learned_addr learned my addr 172.21.15.45:0/2513981754 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 -- 172.21.15.45:0/2513981754 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 912618882 0 0) 0x7fe05c09d9e0 con 0x7fe05c0a4d20 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient(hunting): my global_id is 174174 2024-09-16T03:22:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 -- 172.21.15.45:0/2513981754 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe048003530 con 0x7fe05c0a4d20 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 -- 172.21.15.45:0/2513981754 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1673749926 0 0) 0x7fe048003530 con 0x7fe05c0a4d20 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 -- 172.21.15.45:0/2513981754 >> v1:172.21.15.45:6790/0 conn(0x7fe05c073d40 legacy=0x7fe05c140660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 -- 172.21.15.45:0/2513981754 >> v1:172.21.15.46:6789/0 conn(0x7fe05c0a5810 legacy=0x7fe05c13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: _finish_hunting 0 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 monclient: found mon.noname-a 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 -- 172.21.15.45:0/2513981754 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05c09cf30 con 0x7fe05c0a4d20 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: _finish_auth 0 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: _check_auth_tickets 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 -- 172.21.15.45:0/2513981754 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe054001a20 con 0x7fe05c0a4d20 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 5 monclient: authenticate success, global_id 174174 2024-09-16T03:22:19.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: dump: 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 -- 172.21.15.45:0/2513981754 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe054001c40 con 0x7fe05c0a4d20 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 4 set_mon_vals no callback set 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 1 -- 172.21.15.45:0/2513981754 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe0540020d0 con 0x7fe05c0a4d20 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe060a2e640 10 monclient: dump: 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.507 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: shutdown 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 -- 172.21.15.45:0/2513981754 >> v1:172.21.15.45:6789/0 conn(0x7fe05c0a4d20 legacy=0x7fe05c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 -- 172.21.15.45:0/2513981754 shutdown_connections 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 -- 172.21.15.45:0/2513981754 >> 172.21.15.45:0/2513981754 conn(0x7fe05c0aa720 msgr2=0x7fe05c0aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 -- 172.21.15.45:0/2513981754 shutdown_connections 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 1 -- 172.21.15.45:0/2513981754 wait complete. 2024-09-16T03:22:19.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: build_initial_monmap 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.492+0000 7fe063c94640 10 monclient: monmap: 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 librados: starting msgr at 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 librados: starting objecter 2024-09-16T03:22:19.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 Processor -- start 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 -- start start 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 librados: setting wanted keys 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 librados: calling monclient init 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient: init 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient(hunting): picked mon.noname-a con 0x7fe05c073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient(hunting): picked mon.noname-b con 0x7fe05c0a4d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient(hunting): picked mon.noname-c con 0x7fe05c0a5810 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c140f60 con 0x7fe05c073d40 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c140200 con 0x7fe05c0a4d20 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c1404e0 con 0x7fe05c0a5810 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:19.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient(hunting): authenticate will time out at 2030.148912s 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe062231640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7fe05c0a5810 legacy=0x7fe05c1d4fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe061a30640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe05c073d40 0x7fe05c13c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58524/0 (socket says 172.21.15.45:58524) 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe061a30640 1 -- 172.21.15.45:0/335739528 learned_addr learned my addr 172.21.15.45:0/335739528 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 client.?.objecter ms_handle_connect 0x7fe05c073d40 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2945177433 0 0) 0x7fe05c09cf30 con 0x7fe05c073d40 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): my global_id is 174177 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe040004ec0 con 0x7fe05c073d40 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 client.?.objecter ms_handle_connect 0x7fe05c0a4d20 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:19.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4062612111 0 0) 0x7fe040004ec0 con 0x7fe05c073d40 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe05c09cf30 con 0x7fe05c073d40 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe054001f90 con 0x7fe05c073d40 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): dump: 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3943620574 0 0) 0x7fe05c140200 con 0x7fe05c0a4d20 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): my global_id is 164266 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe040004ec0 con 0x7fe05c0a4d20 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1759872710 0 0) 0x7fe05c09cf30 con 0x7fe05c073d40 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 >> v1:172.21.15.45:6790/0 conn(0x7fe05c0a5810 legacy=0x7fe05c1d4fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 >> v1:172.21.15.46:6789/0 conn(0x7fe05c0a4d20 legacy=0x7fe05c13d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: _finish_hunting 0 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 monclient: found mon.a 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05c0a1d80 con 0x7fe05c073d40 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: _finish_auth 0 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: _check_auth_tickets 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 5 monclient: authenticate success, global_id 174177 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient: _renew_subs 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe05c13ff20 con 0x7fe05c073d40 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 client.174177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient: _renew_subs 2024-09-16T03:22:19.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe05c13d740 con 0x7fe05c073d40 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 librados: init done 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe054001ba0 con 0x7fe05c073d40 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe053fff640 4 set_mon_vals no callback set 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe0540051a0 con 0x7fe05c073d40 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: dump: 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.515 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe0440065d0 con 0x7fe044002870 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 client.174177.objecter ms_handle_connect 0x7fe044002870 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 client.174177.objecter resend_mon_ops 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fe054011210 con 0x7fe05c073d40 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(697..697 src has 99..697) ==== 5930+0+0 (unknown 3213671683 0 0) 0x7fe05404be10 con 0x7fe05c073d40 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 client.174177.objecter ms_dispatch 0x7fe05c000ba0 osd_map(697..697 src has 99..697) 2024-09-16T03:22:19.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 3 client.174177.objecter handle_osd_map got epochs [697,697] > 0 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 3 client.174177.objecter handle_osd_map decoding full epoch 697 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 20 client.174177.objecter dump_active .. 0 homeless 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302637348 0 0) 0x7fe0440065d0 con 0x7fe044002870 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient(hunting): my global_id is 174180 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe05c09cf30 con 0x7fe044002870 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3576304299 0 0) 0x7fe05c09cf30 con 0x7fe044002870 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0440065d0 con 0x7fe044002870 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe04c001820 con 0x7fe044002870 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 monclient: dump: 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe062231640 10 monclient: get_auth_request con 0x7fe04003dfb0 auth_method 0 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2964082840 0 0) 0x7fe0440065d0 con 0x7fe044002870 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe04400a170 con 0x7fe044002870 2024-09-16T03:22:19.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.496+0000 7fe0527fc640 10 client.174177.objecter ms_handle_connect 0x7fe04003dfb0 2024-09-16T03:22:19.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.500+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe04c001a00 con 0x7fe044002870 2024-09-16T03:22:19.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.500+0000 7fe0527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:19.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.500+0000 7fe0527fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:19.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.500+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 >> v1:172.21.15.45:6789/0 conn(0x7fe044002870 legacy=0x7fe044004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.500+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 >> v1:172.21.15.45:6789/0 conn(0x7fe044002870 legacy=0x7fe044004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe044009e80 con 0x7fe044009140 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 client.174177.objecter ms_handle_connect 0x7fe044009140 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 client.174177.objecter resend_mon_ops 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3415096494 0 0) 0x7fe044009e80 con 0x7fe044009140 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 monclient(hunting): my global_id is 174183 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0440065d0 con 0x7fe044009140 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3317605397 0 0) 0x7fe0440065d0 con 0x7fe044009140 2024-09-16T03:22:19.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe044009e80 con 0x7fe044009140 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fe04c0037a0 con 0x7fe044009140 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 monclient: dump: 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.522 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 470700433 0 0) 0x7fe044009e80 con 0x7fe044009140 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe04400c400 con 0x7fe044009140 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4125 (unknown 0 0 709511931) 0x7fe04c0037a0 con 0x7fe044009140 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 >> v1:172.21.15.45:6789/0 conn(0x7fe044009140 legacy=0x7fe0440095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe0527fc640 1 -- 172.21.15.45:0/335739528 >> v1:172.21.15.45:6789/0 conn(0x7fe044009140 legacy=0x7fe0440095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:19.523 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 74, 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 17763, 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: 1 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:19.524 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:22:19.525 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:22:19.526 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:22:19.527 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.528 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.529 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:19.530 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:19.531 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.532 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:19.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 6 2024-09-16T03:22:19.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.535 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.535 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:22:19.535 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:22:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 10 librados: watch_flush enter 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 10 librados: watch_flush exit 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 20 client.174177.objecter shutdown clearing up homeless session... 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 10 client.174177.objecter successfully canceled tick 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 >> v1:172.21.15.46:6812/149360280 conn(0x7fe04003dfb0 legacy=0x7fe040040470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 10 monclient: shutdown 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 >> v1:172.21.15.45:6789/0 conn(0x7fe05c073d40 legacy=0x7fe05c13c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe062231640 1 -- 172.21.15.45:0/335739528 reap_dead start 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe062231640 1 -- 172.21.15.45:0/335739528 reap_dead start 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 shutdown_connections 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 >> 172.21.15.45:0/335739528 conn(0x7fe05c0aa720 msgr2=0x7fe05c0a34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 shutdown_connections 2024-09-16T03:22:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 1 -- 172.21.15.45:0/335739528 wait complete. 2024-09-16T03:22:19.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.504+0000 7fe063c94640 1 librados: shutdown 2024-09-16T03:22:19.538 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:22:19.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:22:19.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:22:19.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient: get_monmap_and_config 2024-09-16T03:22:19.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient: build_initial_monmap 2024-09-16T03:22:19.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient: monmap: 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:22:19.656582+0000 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:22:19.656582+0000 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:22:19.655 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 Processor -- start 2024-09-16T03:22:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 -- start start 2024-09-16T03:22:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient: init 2024-09-16T03:22:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:22:19.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient(hunting): picked mon.noname-b con 0x7f169009b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient(hunting): picked mon.noname-c con 0x7f169009b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient(hunting): picked mon.noname-a con 0x7f169013b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169009cc20 con 0x7f169013b000 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169009d820 con 0x7f169009b250 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169009b6a0 con 0x7f169009b8e0 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient(hunting): authenticate will time out at 2030.305812s 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1695f35640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f169009b8e0 legacy=0x7f169009bcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1696736640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f169013b000 0x7f169013d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58546/0 (socket says 172.21.15.45:58546) 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1696736640 1 -- 172.21.15.45:0/4183464105 learned_addr learned my addr 172.21.15.45:0/4183464105 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 -- 172.21.15.45:0/4183464105 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1240421466 0 0) 0x7f169009cc20 con 0x7f169013b000 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient(hunting): my global_id is 174186 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 -- 172.21.15.45:0/4183464105 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f167c003530 con 0x7f169013b000 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 -- 172.21.15.45:0/4183464105 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 474289010 0 0) 0x7f167c003530 con 0x7f169013b000 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 -- 172.21.15.45:0/4183464105 >> v1:172.21.15.45:6790/0 conn(0x7f169009b8e0 legacy=0x7f169009bcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 -- 172.21.15.45:0/4183464105 >> v1:172.21.15.46:6789/0 conn(0x7f169009b250 legacy=0x7f16901398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: _finish_hunting 0 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 monclient: found mon.noname-a 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 -- 172.21.15.45:0/4183464105 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16900a21f0 con 0x7f169013b000 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: _finish_auth 0 2024-09-16T03:22:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: _check_auth_tickets 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 -- 172.21.15.45:0/4183464105 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f16880016a0 con 0x7f169013b000 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 5 monclient: authenticate success, global_id 174186 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: dump: 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 -- 172.21.15.45:0/4183464105 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1688001840 con 0x7f169013b000 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 1 -- 172.21.15.45:0/4183464105 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1688001c50 con 0x7f169013b000 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1694f33640 10 monclient: dump: 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient: get_monmap_and_config success 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 4 set_mon_vals no callback set 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 10 monclient: shutdown 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 -- 172.21.15.45:0/4183464105 >> v1:172.21.15.45:6789/0 conn(0x7f169013b000 legacy=0x7f169013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 -- 172.21.15.45:0/4183464105 shutdown_connections 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 -- 172.21.15.45:0/4183464105 >> 172.21.15.45:0/4183464105 conn(0x7f16900a2bd0 msgr2=0x7f16900a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 -- 172.21.15.45:0/4183464105 shutdown_connections 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.652+0000 7f1698199640 1 -- 172.21.15.45:0/4183464105 wait complete. 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient: build_initial_monmap 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient: monmap: 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:22:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 librados: starting msgr at 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 librados: starting objecter 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 Processor -- start 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 -- start start 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 librados: setting wanted keys 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 librados: calling monclient init 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient: init 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient: _reopen_session rank -1 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient(hunting): picked mon.noname-b con 0x7f169009b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient(hunting): picked mon.noname-c con 0x7f169009b8e0 addr v1:172.21.15.45:6790/0 2024-09-16T03:22:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient(hunting): picked mon.noname-a con 0x7f169013b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169013e900 con 0x7f169013b000 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169013aa20 con 0x7f169009b250 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169009d2c0 con 0x7f169009b8e0 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient(hunting): _renew_subs 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient(hunting): authenticate will time out at 2030.309059s 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1695734640 1 -- >> v1:172.21.15.45:6790/0 conn(0x7f169009b8e0 legacy=0x7f1690139760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.45:6790/0 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1696736640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f169013b000 0x7f1690139e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58554/0 (socket says 172.21.15.45:58554) 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1696736640 1 -- 172.21.15.45:0/1370572974 learned_addr learned my addr 172.21.15.45:0/1370572974 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 client.?.objecter ms_handle_connect 0x7f169013b000 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583695877 0 0) 0x7f169013e900 con 0x7f169013b000 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): my global_id is 174189 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1674004ef0 con 0x7f169013b000 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 client.?.objecter ms_handle_connect 0x7f169009b250 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1771163586 0 0) 0x7f1674004ef0 con 0x7f169013b000 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f169013e900 con 0x7f169013b000 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1688001b90 con 0x7f169013b000 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): dump: 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 914346210 0 0) 0x7f169013aa20 con 0x7f169009b250 2024-09-16T03:22:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): my global_id is 164272 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1674004ef0 con 0x7f169009b250 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2992893235 0 0) 0x7f169013e900 con 0x7f169013b000 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 >> v1:172.21.15.45:6790/0 conn(0x7f169009b8e0 legacy=0x7f1690139760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 >> v1:172.21.15.46:6789/0 conn(0x7f169009b250 legacy=0x7f1690139050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient: _finish_hunting 0 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 monclient: found mon.a 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:22:19.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1690070110 con 0x7f169013b000 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient: _finish_auth 0 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient: _check_auth_tickets 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 5 monclient: authenticate success, global_id 174189 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient: _renew_subs 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f169006e690 con 0x7f169013b000 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 client.174189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient: _renew_subs 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f16901d3ae0 con 0x7f169013b000 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 librados: init done 2024-09-16T03:22:19.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1688001820 con 0x7f169013b000 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1688004da0 con 0x7f169013b000 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient: dump: 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.676 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1687fff640 4 set_mon_vals no callback set 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f1688010e10 con 0x7f169013b000 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f16780065d0 con 0x7f1678002870 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(697..697 src has 99..697) ==== 5930+0+0 (unknown 3213671683 0 0) 0x7f168804b9e0 con 0x7f169013b000 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 client.174189.objecter ms_dispatch 0x7f1690000ba0 osd_map(697..697 src has 99..697) 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 3 client.174189.objecter handle_osd_map got epochs [697,697] > 0 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 3 client.174189.objecter handle_osd_map decoding full epoch 697 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 20 client.174189.objecter dump_active .. 0 homeless 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 client.174189.objecter ms_handle_connect 0x7f1678002870 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 client.174189.objecter resend_mon_ops 2024-09-16T03:22:19.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f1695734640 10 monclient: get_auth_request con 0x7f167403e090 auth_method 0 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3585663220 0 0) 0x7f16780065d0 con 0x7f1678002870 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 monclient(hunting): my global_id is 164275 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f169013e900 con 0x7f1678002870 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.656+0000 7f16867fc640 10 client.174189.objecter ms_handle_connect 0x7f167403e090 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 399355718 0 0) 0x7f169013e900 con 0x7f1678002870 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16780065d0 con 0x7f1678002870 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f168c0044b0 con 0x7f1678002870 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:22:19.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 10 monclient: dump: 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2866914492 0 0) 0x7f16780065d0 con 0x7f1678002870 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f167800a170 con 0x7f1678002870 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f168c006770 con 0x7f1678002870 2024-09-16T03:22:19.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:22:19.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:22:19.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 >> v1:172.21.15.46:6789/0 conn(0x7f1678002870 legacy=0x7f1678004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.660+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 >> v1:172.21.15.46:6789/0 conn(0x7f1678002870 legacy=0x7f1678004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:19.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.664+0000 7f1698199640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:22:19.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.664+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1678009e60 con 0x7f1678009120 2024-09-16T03:22:19.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 client.174189.objecter ms_handle_connect 0x7f1678009120 2024-09-16T03:22:19.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 client.174189.objecter resend_mon_ops 2024-09-16T03:22:19.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3948409458 0 0) 0x7f1678009e60 con 0x7f1678009120 2024-09-16T03:22:19.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 monclient(hunting): _negotiate 2024-09-16T03:22:19.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:22:19.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:22:19.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 monclient(hunting): my global_id is 164278 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16780065d0 con 0x7f1678009120 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 840592391 0 0) 0x7f16780065d0 con 0x7f1678009120 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1678009e60 con 0x7f1678009120 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f168c005f00 con 0x7f1678009120 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 monclient: dump: 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:22:19.682 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1809519316 0 0) 0x7f1678009e60 con 0x7f1678009120 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f167800c400 con 0x7f1678009120 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4291 (unknown 0 0 1782935997) 0x7f168c006920 con 0x7f1678009120 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 >> v1:172.21.15.46:6789/0 conn(0x7f1678009120 legacy=0x7f16780095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f16867fc640 1 -- 172.21.15.45:0/1370572974 >> v1:172.21.15.46:6789/0 conn(0x7f1678009120 legacy=0x7f16780095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:22:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 74, 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 141978, 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: 1 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:22:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:22:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:22:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:22:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:22:19.695 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:22:19.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 10 librados: watch_flush enter 2024-09-16T03:22:19.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 10 librados: watch_flush exit 2024-09-16T03:22:19.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 20 client.174189.objecter shutdown clearing up homeless session... 2024-09-16T03:22:19.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 10 client.174189.objecter successfully canceled tick 2024-09-16T03:22:19.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 >> v1:172.21.15.46:6812/149360280 conn(0x7f167403e090 legacy=0x7f1674040550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 10 monclient: shutdown 2024-09-16T03:22:19.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:22:19.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 >> v1:172.21.15.45:6789/0 conn(0x7f169013b000 legacy=0x7f1690139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:22:19.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1696736640 1 -- 172.21.15.45:0/1370572974 reap_dead start 2024-09-16T03:22:19.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1696736640 1 -- 172.21.15.45:0/1370572974 reap_dead start 2024-09-16T03:22:19.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 shutdown_connections 2024-09-16T03:22:19.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 >> 172.21.15.45:0/1370572974 conn(0x7f16900a2bd0 msgr2=0x7f169009f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:22:19.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 shutdown_connections 2024-09-16T03:22:19.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 1 -- 172.21.15.45:0/1370572974 wait complete. 2024-09-16T03:22:19.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:22:19.668+0000 7f1698199640 1 librados: shutdown 2024-09-16T03:22:19.698 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:22:19.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T03:22:24.102 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8010 ms) 2024-09-16T03:22:24.102 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-16T03:22:24.102 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13087 ms) 2024-09-16T03:22:24.102 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-16T03:22:24.102 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8059 ms) 2024-09-16T03:22:24.102 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-16T03:22:24.102 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12883 ms) 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8067 ms) 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (22922 ms) 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (23221 ms) 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,0 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: first is 1726456844 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,0 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,0 2024-09-16T03:22:24.103 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,1726456846,1726456847,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,1726456846,1726456847,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,1726456846,1726456847,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,1726456846,1726456847,1726456849,1726456850,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,1726456846,1726456847,1726456849,1726456850,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,1726456846,1726456847,1726456849,1726456850,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,1726456846,1726456847,1726456849,1726456850,1726456852,1726456853,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,1726456846,1726456847,1726456849,1726456850,1726456852,1726456853,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456844,1726456846,1726456847,1726456849,1726456850,1726456852,1726456853,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: got ls 1726456847,1726456849,1726456850,1726456852,1726456853,1726456855,1726456856,0 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: first now 1726456847, trimmed 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (19730 ms) 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-16T03:22:24.104 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: foo0 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13909 ms) 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17961 ms) 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23040 ms) 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4973 ms) 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3020 ms) 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-16T03:22:24.105 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3181 ms) 2024-09-16T03:22:24.106 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-16T03:22:24.106 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (18102 ms) 2024-09-16T03:22:24.106 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (296159 ms total) 2024-09-16T03:22:24.106 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: 2024-09-16T03:22:24.106 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-16T03:22:24.106 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (982525 ms total) 2024-09-16T03:22:24.106 INFO:tasks.workunit.client.0.smithi046.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-16T03:22:24.109 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28525 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28525 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28696 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28696 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29082 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29082 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28944 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28944 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28780 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28780 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28620 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28620 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28895 2024-09-16T03:22:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28895 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29285 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29285 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29326 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29326 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28635 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28635 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.112 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29118 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29118 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28509 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28509 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28541 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28541 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29053 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29053 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28501 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28501 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28517 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28517 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29434 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29434 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28734 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28734 2024-09-16T03:22:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29501 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29501 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28828 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28828 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29253 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29253 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29525 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29525 2024-09-16T03:22:24.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28748 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28748 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29069 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29069 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29389 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29389 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29181 2024-09-16T03:22:24.116 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29181 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29469 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29469 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29006 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29006 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28669 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28669 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29517 2024-09-16T03:22:24.118 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29517 2024-09-16T03:22:24.118 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28952 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28952 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28493 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28493 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28533 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28533 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28930 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28930 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28560 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28560 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28970 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28970 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=28989 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 28989 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29362 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29362 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ pid=29539 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ wait 29539 2024-09-16T03:22:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ exit 0 2024-09-16T03:22:24.122 INFO:tasks.workunit.client.0.smithi046.stderr:+ cleanup 2024-09-16T03:22:24.122 INFO:tasks.workunit.client.0.smithi046.stderr:+ pkill -P 28487 2024-09-16T03:22:24.123 INFO:tasks.workunit.client.0.smithi046.stderr:+ true 2024-09-16T03:22:24.123 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T03:22:24.123 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T03:22:24.179 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-09-16T03:22:24.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T03:22:24.543 DEBUG:teuthology.parallel:result is None 2024-09-16T03:22:24.543 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T03:22:24.556 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T03:22:24.556 DEBUG:teuthology.orchestra.run.smithi046:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T03:22:24.613 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T03:22:24.613 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T03:22:24.622 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T03:23:49.699 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:23:49.699 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:23:49.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:23:49.703 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:23:49.704 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:23:49.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:23:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: get_monmap_and_config 2024-09-16T03:23:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: build_initial_monmap 2024-09-16T03:23:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: monmap: 2024-09-16T03:23:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:49.836562+0000 2024-09-16T03:23:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:49.836562+0000 2024-09-16T03:23:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:49.835 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:49.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 Processor -- start 2024-09-16T03:23:49.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 -- start start 2024-09-16T03:23:49.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: init 2024-09-16T03:23:49.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff29009b8e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff29009b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff29013b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29009cc20 con 0x7ff29013b000 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29009d820 con 0x7ff29009b8e0 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29009b6a0 con 0x7ff29009b250 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient(hunting): authenticate will time out at 2120.485482s 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff294c61640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff29009b250 0x7ff2901398e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39656/0 (socket says 172.21.15.45:39656) 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff295c63640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff29013b000 0x7ff29013d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48252/0 (socket says 172.21.15.45:48252) 2024-09-16T03:23:49.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff294c61640 1 -- 172.21.15.45:0/2564069821 learned_addr learned my addr 172.21.15.45:0/2564069821 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 -- 172.21.15.45:0/2564069821 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368420027 0 0) 0x7ff29009cc20 con 0x7ff29013b000 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient(hunting): my global_id is 174192 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 -- 172.21.15.45:0/2564069821 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff27c003530 con 0x7ff29013b000 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 -- 172.21.15.45:0/2564069821 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1327478042 0 0) 0x7ff27c003530 con 0x7ff29013b000 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 -- 172.21.15.45:0/2564069821 >> v1:172.21.15.45:6790/0 conn(0x7ff29009b250 legacy=0x7ff2901398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 -- 172.21.15.45:0/2564069821 >> v1:172.21.15.46:6789/0 conn(0x7ff29009b8e0 legacy=0x7ff29009bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: _finish_hunting 0 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 monclient: found mon.noname-a 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 -- 172.21.15.45:0/2564069821 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2900a21f0 con 0x7ff29013b000 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: _finish_auth 0 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: _check_auth_tickets 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 -- 172.21.15.45:0/2564069821 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff288001a20 con 0x7ff29013b000 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 5 monclient: authenticate success, global_id 174192 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: dump: 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:49.844 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 -- 172.21.15.45:0/2564069821 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff288001c40 con 0x7ff29013b000 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 1 -- 172.21.15.45:0/2564069821 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff288002010 con 0x7ff29013b000 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:49.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff287fff640 10 monclient: dump: 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 4 set_mon_vals no callback set 2024-09-16T03:23:49.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: shutdown 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 -- 172.21.15.45:0/2564069821 >> v1:172.21.15.45:6789/0 conn(0x7ff29013b000 legacy=0x7ff29013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 -- 172.21.15.45:0/2564069821 shutdown_connections 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 -- 172.21.15.45:0/2564069821 >> 172.21.15.45:0/2564069821 conn(0x7ff2900a2bd0 msgr2=0x7ff2900a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 -- 172.21.15.45:0/2564069821 shutdown_connections 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 -- 172.21.15.45:0/2564069821 wait complete. 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: build_initial_monmap 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 10 monclient: monmap: 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 librados: starting msgr at 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 librados: starting objecter 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.835+0000 7ff2976c6640 1 Processor -- start 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 -- start start 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 librados: setting wanted keys 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 librados: calling monclient init 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient: init 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff29009b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff29009b8e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff29013b000 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29009d2c0 con 0x7ff29013b000 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff290094930 con 0x7ff29009b8e0 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29013a490 con 0x7ff29009b250 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient(hunting): authenticate will time out at 2120.488508s 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff295c63640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff29013b000 0x7ff2901d5550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48268/0 (socket says 172.21.15.45:48268) 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff295462640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff29009b250 0x7ff2901ccdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39658/0 (socket says 172.21.15.45:39658) 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff295c63640 1 -- 172.21.15.45:0/1966252415 learned_addr learned my addr 172.21.15.45:0/1966252415 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 client.?.objecter ms_handle_connect 0x7ff29013b000 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:49.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 client.?.objecter ms_handle_connect 0x7ff29009b250 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 85462934 0 0) 0x7ff2900a21f0 con 0x7ff29013b000 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient(hunting): my global_id is 174195 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff274004ec0 con 0x7ff29013b000 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 client.?.objecter ms_handle_connect 0x7ff29009b8e0 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3276161914 0 0) 0x7ff274004ec0 con 0x7ff29013b000 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2900a21f0 con 0x7ff29013b000 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff288001f90 con 0x7ff29013b000 2024-09-16T03:23:49.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient(hunting): dump: 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:49.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 114529434 0 0) 0x7ff2900a21f0 con 0x7ff29013b000 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 >> v1:172.21.15.45:6790/0 conn(0x7ff29009b250 legacy=0x7ff2901ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 >> v1:172.21.15.46:6789/0 conn(0x7ff29009b8e0 legacy=0x7ff2901d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 monclient: found mon.a 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29013dc70 con 0x7ff29013b000 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient: _finish_auth 0 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 5 monclient: authenticate success, global_id 174195 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient: _renew_subs 2024-09-16T03:23:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff290094650 con 0x7ff29013b000 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 client.174195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient: _renew_subs 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff290091ba0 con 0x7ff29013b000 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 librados: init done 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff288001ba0 con 0x7ff29013b000 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff288005060 con 0x7ff29013b000 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2877fe640 4 set_mon_vals no callback set 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 monclient: dump: 2024-09-16T03:23:49.853 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:49.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7ff288005200 con 0x7ff29013b000 2024-09-16T03:23:49.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff270005090 con 0x7ff29013b000 2024-09-16T03:23:49.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7ff28804b790 con 0x7ff29013b000 2024-09-16T03:23:49.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 client.174195.objecter ms_dispatch 0x7ff290000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:49.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 3 client.174195.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:49.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 3 client.174195.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:49.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 20 client.174195.objecter dump_active .. 0 homeless 2024-09-16T03:23:49.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff295462640 10 monclient: get_auth_request con 0x7ff27403dfe0 auth_method 0 2024-09-16T03:23:49.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.839+0000 7ff285ffb640 10 client.174195.objecter ms_handle_connect 0x7ff27403dfe0 2024-09-16T03:23:49.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.843+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7ff29009d2c0 con 0x7ff29013b000 2024-09-16T03:23:49.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.843+0000 7ff285ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:49.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:49.843+0000 7ff285ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:50.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:23:50.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:23:50.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff270005380 con 0x7ff29013b000 2024-09-16T03:23:50.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff285ffb640 1 -- 172.21.15.45:0/1966252415 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 3342387668) 0x7ff2880167a0 con 0x7ff29013b000 2024-09-16T03:23:50.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff285ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:23:50.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff285ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:23:50.155 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:23:50.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 10 librados: watch_flush enter 2024-09-16T03:23:50.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 10 librados: watch_flush exit 2024-09-16T03:23:50.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 20 client.174195.objecter shutdown clearing up homeless session... 2024-09-16T03:23:50.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 10 client.174195.objecter successfully canceled tick 2024-09-16T03:23:50.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 >> v1:172.21.15.46:6812/149360280 conn(0x7ff27403dfe0 legacy=0x7ff2740404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 10 monclient: shutdown 2024-09-16T03:23:50.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:50.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 >> v1:172.21.15.45:6789/0 conn(0x7ff29013b000 legacy=0x7ff2901d5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 shutdown_connections 2024-09-16T03:23:50.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 >> 172.21.15.45:0/1966252415 conn(0x7ff2900a2bd0 msgr2=0x7ff2900a5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:50.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 shutdown_connections 2024-09-16T03:23:50.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 1 -- 172.21.15.45:0/1966252415 wait complete. 2024-09-16T03:23:50.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.155+0000 7ff2976c6640 1 librados: shutdown 2024-09-16T03:23:50.174 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:23:50.174 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:23:50.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:23:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient: get_monmap_and_config 2024-09-16T03:23:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient: build_initial_monmap 2024-09-16T03:23:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient: monmap: 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:50.289845+0000 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:50.289845+0000 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:50.288 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 1 Processor -- start 2024-09-16T03:23:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 1 -- start start 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient: init 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient(hunting): picked mon.noname-c con 0x7f1fb80a2480 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient(hunting): picked mon.noname-a con 0x7f1fb80a1fa0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient(hunting): picked mon.noname-b con 0x7f1fb81424f0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fb809c7d0 con 0x7f1fb80a1fa0 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fb809bdf0 con 0x7f1fb80a2480 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fb813d0c0 con 0x7f1fb81424f0 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fbd915640 10 monclient(hunting): authenticate will time out at 2120.939183s 2024-09-16T03:23:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fb67fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1fb80a1fa0 0x7f1fb813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48276/0 (socket says 172.21.15.45:48276) 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fb77fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1fb81424f0 0x7f1fb81448e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39664/0 (socket says 172.21.15.45:39664) 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fb67fc640 1 -- 172.21.15.45:0/272108903 learned_addr learned my addr 172.21.15.45:0/272108903 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993300963 0 0) 0x7f1fa0001120 con 0x7f1fb80a1fa0 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fb5ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fb5ffb640 10 monclient(hunting): my global_id is 174198 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.287+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fa4003530 con 0x7f1fb80a1fa0 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1546023457 0 0) 0x7f1fa8001120 con 0x7f1fb80a2480 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient(hunting): my global_id is 164281 2024-09-16T03:23:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fa0001120 con 0x7f1fb80a2480 2024-09-16T03:23:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3382093231 0 0) 0x7f1fa4003530 con 0x7f1fb80a1fa0 2024-09-16T03:23:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 >> v1:172.21.15.45:6790/0 conn(0x7f1fb81424f0 legacy=0x7f1fb81448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 >> v1:172.21.15.46:6789/0 conn(0x7f1fb80a2480 legacy=0x7f1fb813c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:23:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 monclient: found mon.noname-a 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb809ca50 con 0x7f1fb80a1fa0 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: _finish_auth 0 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 5 monclient: authenticate success, global_id 174198 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fa0001870 con 0x7f1fb80a1fa0 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: dump: 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.295 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fa0001b00 con 0x7f1fb80a1fa0 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 4 set_mon_vals no callback set 2024-09-16T03:23:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 1 -- 172.21.15.45:0/272108903 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fa00048b0 con 0x7f1fb80a1fa0 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb5ffb640 10 monclient: dump: 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: shutdown 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- 172.21.15.45:0/272108903 >> v1:172.21.15.45:6789/0 conn(0x7f1fb80a1fa0 legacy=0x7f1fb813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- 172.21.15.45:0/272108903 shutdown_connections 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- 172.21.15.45:0/272108903 >> 172.21.15.45:0/272108903 conn(0x7f1fb80a2bd0 msgr2=0x7f1fb80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- 172.21.15.45:0/272108903 shutdown_connections 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- 172.21.15.45:0/272108903 wait complete. 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: build_initial_monmap 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: monmap: 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:50.298 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 librados: starting msgr at 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 librados: starting objecter 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 Processor -- start 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- start start 2024-09-16T03:23:50.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 librados: setting wanted keys 2024-09-16T03:23:50.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 librados: calling monclient init 2024-09-16T03:23:50.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: init 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient(hunting): picked mon.noname-c con 0x7f1fb80a2480 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient(hunting): picked mon.noname-a con 0x7f1fb81424f0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient(hunting): picked mon.noname-b con 0x7f1fb813efb0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fb8141680 con 0x7f1fb81424f0 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fb8141960 con 0x7f1fb813efb0 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fb8145990 con 0x7f1fb80a2480 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:50.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient(hunting): authenticate will time out at 2120.942766s 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb67fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1fb81424f0 0x7f1fb81d6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48284/0 (socket says 172.21.15.45:48284) 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb67fc640 1 -- 172.21.15.45:0/1647976434 learned_addr learned my addr 172.21.15.45:0/1647976434 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 client.?.objecter ms_handle_connect 0x7f1fb81424f0 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 client.?.objecter ms_handle_connect 0x7f1fb80a2480 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3250738098 0 0) 0x7f1fb8141680 con 0x7f1fb81424f0 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): my global_id is 174201 2024-09-16T03:23:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f98004ef0 con 0x7f1fb81424f0 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2011882874 0 0) 0x7f1fb8145990 con 0x7f1fb80a2480 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): my global_id is 184100 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fb8141680 con 0x7f1fb80a2480 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 client.?.objecter ms_handle_connect 0x7f1fb813efb0 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4165430577 0 0) 0x7f1fb8141680 con 0x7f1fb80a2480 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fb8145990 con 0x7f1fb80a2480 2024-09-16T03:23:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3570717857 0 0) 0x7f1f98004ef0 con 0x7f1fb81424f0 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fb8141680 con 0x7f1fb81424f0 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fa8001b10 con 0x7f1fb80a2480 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): dump: 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.310 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fa00047f0 con 0x7f1fb81424f0 2024-09-16T03:23:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient(hunting): dump: 2024-09-16T03:23:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.310 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.311 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.311 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.311 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.311 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.312 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.312 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.312 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2935416264 0 0) 0x7f1fb8141680 con 0x7f1fb81424f0 2024-09-16T03:23:50.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 >> v1:172.21.15.45:6790/0 conn(0x7f1fb80a2480 legacy=0x7f1fb813c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 >> v1:172.21.15.46:6789/0 conn(0x7f1fb813efb0 legacy=0x7f1fb81d98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient: _finish_hunting 0 2024-09-16T03:23:50.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 monclient: found mon.a 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb8141c40 con 0x7f1fb81424f0 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient: _finish_auth 0 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient: _check_auth_tickets 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 5 monclient: authenticate success, global_id 174201 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: _renew_subs 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb81413a0 con 0x7f1fb81424f0 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 client.174201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:50.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: _renew_subs 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1fb81dafc0 con 0x7f1fb81424f0 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fbd915640 1 librados: init done 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fa0001b00 con 0x7f1fb81424f0 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fa0004e20 con 0x7f1fb81424f0 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1fb57fa640 4 set_mon_vals no callback set 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.291+0000 7f1faffff640 10 monclient: dump: 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1fbd915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1f9c0065d0 con 0x7f1f9c002870 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f1fa0010e90 con 0x7f1fb81424f0 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 client.174201.objecter ms_handle_connect 0x7f1f9c002870 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 client.174201.objecter resend_mon_ops 2024-09-16T03:23:50.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f1fa004b5e0 con 0x7f1fb81424f0 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 client.174201.objecter ms_dispatch 0x7f1fb8000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 3 client.174201.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 3 client.174201.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 20 client.174201.objecter dump_active .. 0 homeless 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 950007744 0 0) 0x7f1f9c0065d0 con 0x7f1f9c002870 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 monclient(hunting): my global_id is 174204 2024-09-16T03:23:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fb8141680 con 0x7f1f9c002870 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1606728778 0 0) 0x7f1fb8141680 con 0x7f1f9c002870 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f9c0065d0 con 0x7f1f9c002870 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fa8003a70 con 0x7f1f9c002870 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1fb77fe640 10 monclient: get_auth_request con 0x7f1f9803dfc0 auth_method 0 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 monclient: dump: 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3177041846 0 0) 0x7f1f9c0065d0 con 0x7f1f9c002870 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f9c00a170 con 0x7f1f9c002870 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 client.174201.objecter ms_handle_connect 0x7f1f9803dfc0 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1fa8003a60 con 0x7f1f9c002870 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 >> v1:172.21.15.45:6789/0 conn(0x7f1f9c002870 legacy=0x7f1f9c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.295+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 >> v1:172.21.15.45:6789/0 conn(0x7f1f9c002870 legacy=0x7f1f9c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.299+0000 7f1fbd915640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:23:50.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.299+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1f9c009e60 con 0x7f1f9c009120 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.299+0000 7f1faffff640 10 client.174201.objecter ms_handle_connect 0x7f1f9c009120 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.299+0000 7f1faffff640 10 client.174201.objecter resend_mon_ops 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 979147317 0 0) 0x7f1f9c009e60 con 0x7f1f9c009120 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 10 monclient(hunting): my global_id is 174207 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f9c0065d0 con 0x7f1f9c009120 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3018497046 0 0) 0x7f1f9c0065d0 con 0x7f1f9c009120 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f9c009e60 con 0x7f1f9c009120 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f1fa8004120 con 0x7f1f9c009120 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 10 monclient: dump: 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3311391922 0 0) 0x7f1f9c009e60 con 0x7f1f9c009120 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 --> v1:172.21.15.45:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1f9c00c3b0 con 0x7f1f9c009120 2024-09-16T03:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 <== mon.0 v1:172.21.15.45:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4137 (unknown 0 0 2395439532) 0x7f1fa80063a0 con 0x7f1f9c009120 2024-09-16T03:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 >> v1:172.21.15.45:6789/0 conn(0x7f1f9c009120 legacy=0x7f1f9c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1faffff640 1 -- 172.21.15.45:0/1647976434 >> v1:172.21.15.45:6789/0 conn(0x7f1f9c009120 legacy=0x7f1f9c0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:23:50.321 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:23:50.321 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:23:50.321 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:23:50.321 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "leader", 2024-09-16T03:23:50.321 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 76, 2024-09-16T03:23:50.322 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 108562, 2024-09-16T03:23:50.322 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:23:50.322 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:23:50.322 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:23:50.322 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:23:50.322 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.322 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:23:50.322 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:23:50.322 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:23:50.323 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:23:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:23:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:23:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:23:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:23:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:23:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:23:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:23:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:23:50.325 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:23:50.326 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:23:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:23:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:23:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:23:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:23:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 5 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.334 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:23:50.334 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:23:50.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 10 librados: watch_flush enter 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 10 librados: watch_flush exit 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 20 client.174201.objecter shutdown clearing up homeless session... 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 10 client.174201.objecter successfully canceled tick 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 >> v1:172.21.15.46:6812/149360280 conn(0x7f1f9803dfc0 legacy=0x7f1f98040480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 10 monclient: shutdown 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 >> v1:172.21.15.45:6789/0 conn(0x7f1fb81424f0 legacy=0x7f1fb81d6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fb77fe640 1 -- 172.21.15.45:0/1647976434 reap_dead start 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fb77fe640 1 -- 172.21.15.45:0/1647976434 reap_dead start 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 shutdown_connections 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 >> 172.21.15.45:0/1647976434 conn(0x7f1fb80a2bd0 msgr2=0x7f1fb8098960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:50.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 shutdown_connections 2024-09-16T03:23:50.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 1 -- 172.21.15.45:0/1647976434 wait complete. 2024-09-16T03:23:50.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.303+0000 7f1fbd915640 1 librados: shutdown 2024-09-16T03:23:50.337 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:23:50.337 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:23:50.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:23:50.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.451+0000 7f23dd347640 10 monclient: get_monmap_and_config 2024-09-16T03:23:50.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.451+0000 7f23dd347640 10 monclient: build_initial_monmap 2024-09-16T03:23:50.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.451+0000 7f23dd347640 10 monclient: monmap: 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:50.455717+0000 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:50.455717+0000 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:50.454 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 Processor -- start 2024-09-16T03:23:50.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- start start 2024-09-16T03:23:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: init 2024-09-16T03:23:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): picked mon.noname-b con 0x7f23d809c570 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): picked mon.noname-c con 0x7f23d8134320 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): picked mon.noname-a con 0x7f23d81347d0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d8135150 con 0x7f23d81347d0 2024-09-16T03:23:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d8074470 con 0x7f23d8134320 2024-09-16T03:23:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d8074750 con 0x7f23d809c570 2024-09-16T03:23:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): authenticate will time out at 2121.104940s 2024-09-16T03:23:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d6ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f23d809c570 0x7f23d809c950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58342/0 (socket says 172.21.15.45:58342) 2024-09-16T03:23:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d6ffd640 1 -- 172.21.15.45:0/1759433992 learned_addr learned my addr 172.21.15.45:0/1759433992 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012725704 0 0) 0x7f23d8074750 con 0x7f23d809c570 2024-09-16T03:23:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient(hunting): my global_id is 184103 2024-09-16T03:23:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23c0003530 con 0x7f23d809c570 2024-09-16T03:23:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923051937 0 0) 0x7f23d00014a0 con 0x7f23d81347d0 2024-09-16T03:23:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient(hunting): my global_id is 174210 2024-09-16T03:23:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23d8074750 con 0x7f23d81347d0 2024-09-16T03:23:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1245972724 0 0) 0x7f23c0003530 con 0x7f23d809c570 2024-09-16T03:23:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 >> v1:172.21.15.46:6789/0 conn(0x7f23d8134320 legacy=0x7f23d813cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 >> v1:172.21.15.45:6789/0 conn(0x7f23d81347d0 legacy=0x7f23d81402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:23:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 monclient: found mon.noname-b 2024-09-16T03:23:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d8074a30 con 0x7f23d809c570 2024-09-16T03:23:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: _finish_auth 0 2024-09-16T03:23:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:23:50.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:50.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 5 monclient: authenticate success, global_id 184103 2024-09-16T03:23:50.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:50.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f23cc0016a0 con 0x7f23d809c570 2024-09-16T03:23:50.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:50.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: dump: 2024-09-16T03:23:50.471 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.471 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.471 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.471 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.472 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.472 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.472 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.472 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.472 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.472 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23cc001840 con 0x7f23d809c570 2024-09-16T03:23:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 1 -- 172.21.15.45:0/1759433992 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f23cc001c50 con 0x7f23d809c570 2024-09-16T03:23:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 4 set_mon_vals no callback set 2024-09-16T03:23:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23d5ffb640 10 monclient: dump: 2024-09-16T03:23:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.475 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: shutdown 2024-09-16T03:23:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- 172.21.15.45:0/1759433992 >> v1:172.21.15.45:6790/0 conn(0x7f23d809c570 legacy=0x7f23d809c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- 172.21.15.45:0/1759433992 shutdown_connections 2024-09-16T03:23:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- 172.21.15.45:0/1759433992 >> 172.21.15.45:0/1759433992 conn(0x7f23d80a2bd0 msgr2=0x7f23d80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- 172.21.15.45:0/1759433992 shutdown_connections 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- 172.21.15.45:0/1759433992 wait complete. 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: build_initial_monmap 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: monmap: 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 librados: starting msgr at 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 librados: starting objecter 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 Processor -- start 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- start start 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 librados: setting wanted keys 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 librados: calling monclient init 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: init 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): picked mon.noname-c con 0x7f23d809c570 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): picked mon.noname-a con 0x7f23d8134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): picked mon.noname-b con 0x7f23d81347d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d8099ae0 con 0x7f23d8134320 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d8133c50 con 0x7f23d81347d0 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d8133f30 con 0x7f23d809c570 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.455+0000 7f23dd347640 10 monclient(hunting): authenticate will time out at 2121.107914s 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23d6ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f23d809c570 0x7f23d81cd010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58356/0 (socket says 172.21.15.45:58356) 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23d6ffd640 1 -- 172.21.15.45:0/3972845380 learned_addr learned my addr 172.21.15.45:0/3972845380 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23d67fc640 1 --1- 172.21.15.45:0/3972845380 >> v1:172.21.15.45:6789/0 conn(0x7f23d8134320 0x7f23d81d1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60188/0 (socket says 172.21.15.45:60188) 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.?.objecter ms_handle_connect 0x7f23d8134320 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.?.objecter ms_handle_connect 0x7f23d809c570 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 598717352 0 0) 0x7f23d8133f30 con 0x7f23d809c570 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): my global_id is 184106 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23c4004ec0 con 0x7f23d809c570 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3172698132 0 0) 0x7f23d8099ae0 con 0x7f23d8134320 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): my global_id is 174213 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23d8133f30 con 0x7f23d8134320 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.?.objecter ms_handle_connect 0x7f23d81347d0 2024-09-16T03:23:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2944533345 0 0) 0x7f23c4004ec0 con 0x7f23d809c570 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23d8099ae0 con 0x7f23d809c570 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f23cc001b90 con 0x7f23d809c570 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): dump: 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 27059036 0 0) 0x7f23d8133f30 con 0x7f23d8134320 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23c4004ec0 con 0x7f23d8134320 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f23c80018c0 con 0x7f23d8134320 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): dump: 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 384615739 0 0) 0x7f23d8133c50 con 0x7f23d81347d0 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): my global_id is 164287 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23d8133f30 con 0x7f23d81347d0 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2626628433 0 0) 0x7f23d8099ae0 con 0x7f23d809c570 2024-09-16T03:23:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 >> v1:172.21.15.46:6789/0 conn(0x7f23d81347d0 legacy=0x7f23d81d5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 >> v1:172.21.15.45:6789/0 conn(0x7f23d8134320 legacy=0x7f23d81d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 monclient: found mon.c 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d809d2c0 con 0x7f23d809c570 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: _finish_auth 0 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 5 monclient: authenticate success, global_id 184106 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 10 monclient: _renew_subs 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f23d809bcc0 con 0x7f23d809c570 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 10 client.184106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 10 monclient: _renew_subs 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f23d8131290 con 0x7f23d809c570 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 1 librados: init done 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23cc004c00 con 0x7f23d809c570 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f23cc004db0 con 0x7f23d809c570 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23d57fa640 4 set_mon_vals no callback set 2024-09-16T03:23:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: dump: 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f23b80065d0 con 0x7f23b8002870 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f23cc010e20 con 0x7f23d809c570 2024-09-16T03:23:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.184106.objecter ms_handle_connect 0x7f23b8002870 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.184106.objecter resend_mon_ops 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f23cc04b470 con 0x7f23d809c570 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.184106.objecter ms_dispatch 0x7f23d8000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 3 client.184106.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 3 client.184106.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 20 client.184106.objecter dump_active .. 0 homeless 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470108419 0 0) 0x7f23b80065d0 con 0x7f23b8002870 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient(hunting): my global_id is 184109 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23d8099ae0 con 0x7f23b8002870 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2719815751 0 0) 0x7f23d8099ae0 con 0x7f23b8002870 2024-09-16T03:23:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23b80065d0 con 0x7f23b8002870 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f23c80037a0 con 0x7f23b8002870 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23d77fe640 10 monclient: get_auth_request con 0x7f23c403e380 auth_method 0 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: dump: 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3296704947 0 0) 0x7f23b80065d0 con 0x7f23b8002870 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23b800a170 con 0x7f23b8002870 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 client.184106.objecter ms_handle_connect 0x7f23c403e380 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f23c80057d0 con 0x7f23b8002870 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 >> v1:172.21.15.45:6790/0 conn(0x7f23b8002870 legacy=0x7f23b8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.459+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 >> v1:172.21.15.45:6790/0 conn(0x7f23b8002870 legacy=0x7f23b8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f23b8009e60 con 0x7f23b8009120 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 client.184106.objecter ms_handle_connect 0x7f23b8009120 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 client.184106.objecter resend_mon_ops 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3724740763 0 0) 0x7f23b8009e60 con 0x7f23b8009120 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 monclient(hunting): my global_id is 184112 2024-09-16T03:23:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23b80065d0 con 0x7f23b8009120 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2765223455 0 0) 0x7f23b80065d0 con 0x7f23b8009120 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23b8009e60 con 0x7f23b8009120 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f23c8003cc0 con 0x7f23b8009120 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 monclient: dump: 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 954200445 0 0) 0x7f23b8009e60 con 0x7f23b8009120 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 --> v1:172.21.15.45:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23b800c3b0 con 0x7f23b8009120 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 <== mon.2 v1:172.21.15.45:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3965 (unknown 0 0 3306692720) 0x7f23c8005980 con 0x7f23b8009120 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 >> v1:172.21.15.45:6790/0 conn(0x7f23b8009120 legacy=0x7f23b80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23bf7fe640 1 -- 172.21.15.45:0/3972845380 >> v1:172.21.15.45:6790/0 conn(0x7f23b8009120 legacy=0x7f23b80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:23:50.493 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 76, 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 683, 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:23:50.494 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:23:50.495 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:23:50.496 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:23:50.497 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:23:50.498 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.499 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:23:50.500 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:23:50.503 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:23:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:23:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:23:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:23:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:23:50.505 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 10 librados: watch_flush enter 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 10 librados: watch_flush exit 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 20 client.184106.objecter shutdown clearing up homeless session... 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 10 client.184106.objecter successfully canceled tick 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 >> v1:172.21.15.46:6812/149360280 conn(0x7f23c403e380 legacy=0x7f23c4040840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 10 monclient: shutdown 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 >> v1:172.21.15.45:6790/0 conn(0x7f23d809c570 legacy=0x7f23d81cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23d77fe640 1 -- 172.21.15.45:0/3972845380 reap_dead start 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23d77fe640 1 -- 172.21.15.45:0/3972845380 reap_dead start 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 shutdown_connections 2024-09-16T03:23:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 >> 172.21.15.45:0/3972845380 conn(0x7f23d80a2bd0 msgr2=0x7f23d80a3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:50.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 shutdown_connections 2024-09-16T03:23:50.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 1 -- 172.21.15.45:0/3972845380 wait complete. 2024-09-16T03:23:50.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.467+0000 7f23dd347640 1 librados: shutdown 2024-09-16T03:23:50.508 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:23:50.508 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:23:50.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:23:50.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient: get_monmap_and_config 2024-09-16T03:23:50.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient: build_initial_monmap 2024-09-16T03:23:50.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient: monmap: 2024-09-16T03:23:50.622 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:50.623 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.623 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:50.624742+0000 2024-09-16T03:23:50.623 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:50.624742+0000 2024-09-16T03:23:50.623 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:50.624 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.624 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:50.624 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:50.624 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:50.624 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 Processor -- start 2024-09-16T03:23:50.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 -- start start 2024-09-16T03:23:50.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient: init 2024-09-16T03:23:50.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:50.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:23:50.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient(hunting): picked mon.noname-a con 0x7f3d6009b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:50.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient(hunting): picked mon.noname-c con 0x7f3d6009b250 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:50.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient(hunting): picked mon.noname-b con 0x7f3d6013b000 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6009cc20 con 0x7f3d6009b8e0 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6009d820 con 0x7f3d6009b250 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6009b6a0 con 0x7f3d6013b000 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient(hunting): authenticate will time out at 2121.273819s 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d65e70640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3d6009b8e0 0x7f3d6009bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60192/0 (socket says 172.21.15.45:60192) 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d66671640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3d6013b000 0x7f3d6013d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58368/0 (socket says 172.21.15.45:58368) 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d65e70640 1 -- 172.21.15.45:0/1067398456 learned_addr learned my addr 172.21.15.45:0/1067398456 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 139913718 0 0) 0x7f3d6009cc20 con 0x7f3d6009b8e0 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient(hunting): my global_id is 174216 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d4c005740 con 0x7f3d6009b8e0 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1829947668 0 0) 0x7f3d6009b6a0 con 0x7f3d6013b000 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient(hunting): my global_id is 184115 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d6009cc20 con 0x7f3d6013b000 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1112090753 0 0) 0x7f3d4c005740 con 0x7f3d6009b8e0 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 >> v1:172.21.15.45:6790/0 conn(0x7f3d6013b000 legacy=0x7f3d6013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 >> v1:172.21.15.46:6789/0 conn(0x7f3d6009b250 legacy=0x7f3d601398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: _finish_hunting 0 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 monclient: found mon.noname-a 2024-09-16T03:23:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:50.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d600a21f0 con 0x7f3d6009b8e0 2024-09-16T03:23:50.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: _finish_auth 0 2024-09-16T03:23:50.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: _check_auth_tickets 2024-09-16T03:23:50.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:50.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3d5c0016a0 con 0x7f3d6009b8e0 2024-09-16T03:23:50.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: dump: 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.637 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 5 monclient: authenticate success, global_id 174216 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d5c001840 con 0x7f3d6009b8e0 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 4 set_mon_vals no callback set 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 1 -- 172.21.15.45:0/1067398456 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3d5c001c50 con 0x7f3d6009b8e0 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d64e6e640 10 monclient: dump: 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 10 monclient: shutdown 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 -- 172.21.15.45:0/1067398456 >> v1:172.21.15.45:6789/0 conn(0x7f3d6009b8e0 legacy=0x7f3d6009bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 -- 172.21.15.45:0/1067398456 shutdown_connections 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 -- 172.21.15.45:0/1067398456 >> 172.21.15.45:0/1067398456 conn(0x7f3d600a2bd0 msgr2=0x7f3d600a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 -- 172.21.15.45:0/1067398456 shutdown_connections 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.623+0000 7f3d680d4640 1 -- 172.21.15.45:0/1067398456 wait complete. 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient: build_initial_monmap 2024-09-16T03:23:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient: monmap: 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 librados: starting msgr at 2024-09-16T03:23:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 librados: starting objecter 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 Processor -- start 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 -- start start 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 librados: setting wanted keys 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 librados: calling monclient init 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient: init 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient(hunting): picked mon.noname-c con 0x7f3d6009b250 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient(hunting): picked mon.noname-a con 0x7f3d6009b8e0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient(hunting): picked mon.noname-b con 0x7f3d6013b000 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d60097460 con 0x7f3d6009b8e0 2024-09-16T03:23:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d600944a0 con 0x7f3d6013b000 2024-09-16T03:23:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6009d2c0 con 0x7f3d6009b250 2024-09-16T03:23:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient(hunting): authenticate will time out at 2121.276991s 2024-09-16T03:23:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d65e70640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3d6009b250 0x7f3d60093680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58382/0 (socket says 172.21.15.45:58382) 2024-09-16T03:23:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d65e70640 1 -- 172.21.15.45:0/2157761116 learned_addr learned my addr 172.21.15.45:0/2157761116 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d6566f640 1 --1- 172.21.15.45:0/2157761116 >> v1:172.21.15.45:6789/0 conn(0x7f3d6009b8e0 0x7f3d60093d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60204/0 (socket says 172.21.15.45:60204) 2024-09-16T03:23:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 client.?.objecter ms_handle_connect 0x7f3d6009b250 2024-09-16T03:23:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 client.?.objecter ms_handle_connect 0x7f3d6009b8e0 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106272476 0 0) 0x7f3d6009d2c0 con 0x7f3d6009b250 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): my global_id is 184118 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d44004ec0 con 0x7f3d6009b250 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2467530718 0 0) 0x7f3d60097460 con 0x7f3d6009b8e0 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): my global_id is 174219 2024-09-16T03:23:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d6009d2c0 con 0x7f3d6009b8e0 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 client.?.objecter ms_handle_connect 0x7f3d6013b000 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1634070135 0 0) 0x7f3d600944a0 con 0x7f3d6013b000 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): my global_id is 164293 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d60097460 con 0x7f3d6013b000 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 172942253 0 0) 0x7f3d6009d2c0 con 0x7f3d6009b8e0 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d600944a0 con 0x7f3d6009b8e0 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3872181883 0 0) 0x7f3d44004ec0 con 0x7f3d6009b250 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d6009d2c0 con 0x7f3d6009b250 2024-09-16T03:23:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3d50001940 con 0x7f3d6009b8e0 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): dump: 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.645 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3d5c001b90 con 0x7f3d6009b250 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): dump: 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 619685767 0 0) 0x7f3d600944a0 con 0x7f3d6009b8e0 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 >> v1:172.21.15.45:6790/0 conn(0x7f3d6009b250 legacy=0x7f3d60093680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 >> v1:172.21.15.46:6789/0 conn(0x7f3d6013b000 legacy=0x7f3d601d5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: _finish_hunting 0 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 monclient: found mon.a 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d600963d0 con 0x7f3d6009b8e0 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: _finish_auth 0 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: _check_auth_tickets 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 5 monclient: authenticate success, global_id 174219 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient: _renew_subs 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d60095100 con 0x7f3d6009b8e0 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 client.174219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient: _renew_subs 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d60095c60 con 0x7f3d6009b8e0 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 librados: init done 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d50001c80 con 0x7f3d6009b8e0 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3d50003a70 con 0x7f3d6009b8e0 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d57fff640 4 set_mon_vals no callback set 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: dump: 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d480065d0 con 0x7f3d48002870 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f3d50003c10 con 0x7f3d6009b8e0 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f3d5004b4f0 con 0x7f3d6009b8e0 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 client.174219.objecter ms_dispatch 0x7f3d60000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 3 client.174219.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 3 client.174219.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 20 client.174219.objecter dump_active .. 0 homeless 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 client.174219.objecter ms_handle_connect 0x7f3d48002870 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 client.174219.objecter resend_mon_ops 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2887069146 0 0) 0x7f3d480065d0 con 0x7f3d48002870 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient(hunting): my global_id is 164296 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d600944a0 con 0x7f3d48002870 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d66671640 10 monclient: get_auth_request con 0x7f3d4403e2c0 auth_method 0 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 816918396 0 0) 0x7f3d600944a0 con 0x7f3d48002870 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d480065d0 con 0x7f3d48002870 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3d5c001820 con 0x7f3d48002870 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.627+0000 7f3d567fc640 10 monclient: dump: 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.631+0000 7f3d567fc640 10 client.174219.objecter ms_handle_connect 0x7f3d4403e2c0 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.631+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1823078417 0 0) 0x7f3d480065d0 con 0x7f3d48002870 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.631+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.46:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d4800a170 con 0x7f3d48002870 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.631+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3d5c004bc0 con 0x7f3d48002870 2024-09-16T03:23:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.631+0000 7f3d567fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:50.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.631+0000 7f3d567fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:50.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.631+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 >> v1:172.21.15.46:6789/0 conn(0x7f3d48002870 legacy=0x7f3d48004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.631+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 >> v1:172.21.15.46:6789/0 conn(0x7f3d48002870 legacy=0x7f3d48004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:23:50.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d680d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:23:50.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d48009e60 con 0x7f3d48009120 2024-09-16T03:23:50.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d567fc640 10 client.174219.objecter ms_handle_connect 0x7f3d48009120 2024-09-16T03:23:50.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d567fc640 10 client.174219.objecter resend_mon_ops 2024-09-16T03:23:50.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83203922 0 0) 0x7f3d48009e60 con 0x7f3d48009120 2024-09-16T03:23:50.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d567fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d567fc640 10 monclient(hunting): my global_id is 164299 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.635+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d480065d0 con 0x7f3d48009120 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1477666834 0 0) 0x7f3d480065d0 con 0x7f3d48009120 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d48009e60 con 0x7f3d48009120 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f3d5c005240 con 0x7f3d48009120 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 10 monclient: dump: 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.655 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1526436273 0 0) 0x7f3d48009e60 con 0x7f3d48009120 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 --> v1:172.21.15.46:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d4800c400 con 0x7f3d48009120 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 <== mon.1 v1:172.21.15.46:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 1903411610) 0x7f3d5c005300 con 0x7f3d48009120 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:23:50.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 >> v1:172.21.15.46:6789/0 conn(0x7f3d48009120 legacy=0x7f3d480095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d567fc640 1 -- 172.21.15.45:0/2157761116 >> v1:172.21.15.46:6789/0 conn(0x7f3d48009120 legacy=0x7f3d480095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "peon", 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: "election_epoch": 76, 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 232949, 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum": [ 2024-09-16T03:23:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_age": 0, 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "required_con": "2450521708007866372", 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "required_mon": [ 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:23:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_con": "4540719139924082687", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "quorum_mon": [ 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:23:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:23:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:23:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:23:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:23:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:23:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:23:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: "outside_quorum": [], 2024-09-16T03:23:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: "extra_probe_peers": [], 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_provider": [], 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": { 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 1, 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "16b8bfd8-03c6-4432-90b7-8e9932b37929", 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-09-16T03:04:14.884162Z", 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release": 19, 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "min_mon_release_name": "squid", 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "election_strategy": 1, 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "disallowed_leaders: ": "", 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false, 2024-09-16T03:23:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: "tiebreaker_mon": "", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_ranks: ": "", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "features": { 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "persistent": [ 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "kraken", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "luminous", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "mimic", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap-prune", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "nautilus", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "elector-pinging", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "reef", 2024-09-16T03:23:50.662 INFO:teuthology.orchestra.run.smithi045.stdout: "squid" 2024-09-16T03:23:50.663 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.663 INFO:teuthology.orchestra.run.smithi045.stdout: "optional": [] 2024-09-16T03:23:50.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.663 INFO:teuthology.orchestra.run.smithi045.stdout: "mons": [ 2024-09-16T03:23:50.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.663 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 0, 2024-09-16T03:23:50.663 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "a", 2024-09-16T03:23:50.663 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789", 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6789/0", 2024-09-16T03:23:50.664 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6789/0", 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 1, 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "b", 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.665 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789", 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: "rank": 2, 2024-09-16T03:23:50.666 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "c", 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790", 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 0 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6790/0", 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6790/0", 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "priority": 0, 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0, 2024-09-16T03:23:50.667 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "{}" 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: "feature_map": { 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": [ 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": [ 2024-09-16T03:23:50.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 3 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: "client": [ 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 2 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-09-16T03:23:50.669 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": [ 2024-09-16T03:23:50.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-09-16T03:23:50.670 INFO:teuthology.orchestra.run.smithi045.stdout: "features": "0x3f03dffffffdffff", 2024-09-16T03:23:50.670 INFO:teuthology.orchestra.run.smithi045.stdout: "release": "squid", 2024-09-16T03:23:50.670 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 1 2024-09-16T03:23:50.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-16T03:23:50.670 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-16T03:23:50.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-16T03:23:50.670 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": false 2024-09-16T03:23:50.670 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 10 librados: watch_flush enter 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 10 librados: watch_flush exit 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 20 client.174219.objecter shutdown clearing up homeless session... 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 10 client.174219.objecter successfully canceled tick 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 >> v1:172.21.15.46:6812/149360280 conn(0x7f3d4403e2c0 legacy=0x7f3d44040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 10 monclient: shutdown 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 >> v1:172.21.15.45:6789/0 conn(0x7f3d6009b8e0 legacy=0x7f3d60093d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d66671640 1 -- 172.21.15.45:0/2157761116 reap_dead start 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d66671640 1 -- 172.21.15.45:0/2157761116 reap_dead start 2024-09-16T03:23:50.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 shutdown_connections 2024-09-16T03:23:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 >> 172.21.15.45:0/2157761116 conn(0x7f3d600a2bd0 msgr2=0x7f3d600a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 shutdown_connections 2024-09-16T03:23:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 1 -- 172.21.15.45:0/2157761116 wait complete. 2024-09-16T03:23:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.639+0000 7f3d680d4640 1 librados: shutdown 2024-09-16T03:23:50.673 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:23:50.673 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:23:50.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:23:50.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient: get_monmap_and_config 2024-09-16T03:23:50.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient: build_initial_monmap 2024-09-16T03:23:50.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient: monmap: 2024-09-16T03:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:50.789747+0000 2024-09-16T03:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:50.789747+0000 2024-09-16T03:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:50.789 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 1 Processor -- start 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 1 -- start start 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient: init 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient(hunting): picked mon.noname-a con 0x7f2298135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient(hunting): picked mon.noname-c con 0x7f229809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient(hunting): picked mon.noname-b con 0x7f229813c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229809f7e0 con 0x7f2298135950 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229809b160 con 0x7f229809fc40 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229809b440 con 0x7f229813c010 2024-09-16T03:23:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f22a0664640 10 monclient(hunting): authenticate will time out at 2121.439196s 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229ec01640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f229813c010 0x7f229813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58390/0 (socket says 172.21.15.45:58390) 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229ec01640 1 -- 172.21.15.45:0/3240691975 learned_addr learned my addr 172.21.15.45:0/3240691975 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2378874573 0 0) 0x7f229809b440 con 0x7f229813c010 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient(hunting): my global_id is 184121 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2284003530 con 0x7f229813c010 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328866630 0 0) 0x7f229809f7e0 con 0x7f2298135950 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient(hunting): my global_id is 174222 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f229809b440 con 0x7f2298135950 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2875453582 0 0) 0x7f2284003530 con 0x7f229813c010 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 >> v1:172.21.15.46:6789/0 conn(0x7f229809fc40 legacy=0x7f229813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 >> v1:172.21.15.45:6789/0 conn(0x7f2298135950 legacy=0x7f2298135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient: _finish_hunting 0 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 monclient: found mon.noname-b 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2298136440 con 0x7f229813c010 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient: _finish_auth 0 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 10 monclient: _check_auth_tickets 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.787+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2290001a20 con 0x7f229813c010 2024-09-16T03:23:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229d3fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229d3fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229d3fe640 10 monclient: dump: 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 5 monclient: authenticate success, global_id 184121 2024-09-16T03:23:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2290001bc0 con 0x7f229813c010 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229d3fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 4 set_mon_vals no callback set 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229d3fe640 1 -- 172.21.15.45:0/3240691975 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2290002010 con 0x7f229813c010 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229d3fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229d3fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229d3fe640 10 monclient: dump: 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: shutdown 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- 172.21.15.45:0/3240691975 >> v1:172.21.15.45:6790/0 conn(0x7f229813c010 legacy=0x7f229813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- 172.21.15.45:0/3240691975 shutdown_connections 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- 172.21.15.45:0/3240691975 >> 172.21.15.45:0/3240691975 conn(0x7f22980a2bd0 msgr2=0x7f22980a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- 172.21.15.45:0/3240691975 shutdown_connections 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- 172.21.15.45:0/3240691975 wait complete. 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: build_initial_monmap 2024-09-16T03:23:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: monmap: 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 librados: starting msgr at 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 librados: starting objecter 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 Processor -- start 2024-09-16T03:23:50.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- start start 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 librados: setting wanted keys 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 librados: calling monclient init 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: init 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient(hunting): picked mon.noname-c con 0x7f229809fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient(hunting): picked mon.noname-b con 0x7f2298135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient(hunting): picked mon.noname-a con 0x7f229813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229813b6f0 con 0x7f229813c010 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229809d2c0 con 0x7f2298135950 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2298073950 con 0x7f229809fc40 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient(hunting): authenticate will time out at 2121.442209s 2024-09-16T03:23:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229ec01640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f229813c010 0x7f22981d54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60216/0 (socket says 172.21.15.45:60216) 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229ec01640 1 -- 172.21.15.45:0/2992567689 learned_addr learned my addr 172.21.15.45:0/2992567689 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 client.?.objecter ms_handle_connect 0x7f229813c010 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 client.?.objecter ms_handle_connect 0x7f229809fc40 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3487133071 0 0) 0x7f2298136440 con 0x7f229813c010 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): my global_id is 174225 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f227c004ec0 con 0x7f229813c010 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 client.?.objecter ms_handle_connect 0x7f2298135950 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97386678 0 0) 0x7f2298073950 con 0x7f229809fc40 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): my global_id is 184124 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2298136440 con 0x7f229809fc40 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4275130254 0 0) 0x7f227c004ec0 con 0x7f229813c010 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2298073950 con 0x7f229813c010 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f2290001f10 con 0x7f229813c010 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): dump: 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 822229387 0 0) 0x7f229809b160 con 0x7f2298135950 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): my global_id is 164305 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f227c004ec0 con 0x7f2298135950 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 62677903 0 0) 0x7f2298136440 con 0x7f229809fc40 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f229809b160 con 0x7f229809fc40 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22880018c0 con 0x7f229809fc40 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient(hunting): dump: 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1352636460 0 0) 0x7f2298073950 con 0x7f229813c010 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 >> v1:172.21.15.45:6790/0 conn(0x7f229809fc40 legacy=0x7f22981ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 >> v1:172.21.15.46:6789/0 conn(0x7f2298135950 legacy=0x7f22981d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient: _finish_hunting 0 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 monclient: found mon.a 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2298073670 con 0x7f229813c010 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient: _finish_auth 0 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient: _check_auth_tickets 2024-09-16T03:23:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 5 monclient: authenticate success, global_id 174225 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: _renew_subs 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2298073c30 con 0x7f229813c010 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 client.174225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: _renew_subs 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2298071030 con 0x7f229813c010 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f22a0664640 1 librados: init done 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2290004f80 con 0x7f229813c010 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f22900052d0 con 0x7f229813c010 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f229cbfd640 4 set_mon_vals no callback set 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.791+0000 7f228effd640 10 monclient: dump: 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f22a0664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f22a0664640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f22a0664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2280005090 con 0x7f229813c010 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f2290011340 con 0x7f229813c010 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f229004b750 con 0x7f229813c010 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f228effd640 10 client.174225.objecter ms_dispatch 0x7f2298000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f228effd640 3 client.174225.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f228effd640 3 client.174225.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f228effd640 20 client.174225.objecter dump_active .. 0 homeless 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f229e400640 10 monclient: get_auth_request con 0x7f227c03e260 auth_method 0 2024-09-16T03:23:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.795+0000 7f228effd640 10 client.174225.objecter ms_handle_connect 0x7f227c03e260 2024-09-16T03:23:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.799+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f229813b6f0 con 0x7f229813c010 2024-09-16T03:23:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.799+0000 7f228effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:50.799+0000 7f228effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f22a0664640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f22a0664640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f22a0664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f2280005c50 con 0x7f229813c010 2024-09-16T03:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f228effd640 1 -- 172.21.15.45:0/2992567689 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f229001ff50 con 0x7f229813c010 2024-09-16T03:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f228effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f228effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:23:51.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=14,logm=46} crc {auth=752244947,config=1097430493,health=951176358,kv=3374818887,logm=3535489320}) 2024-09-16T03:23:51.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=1,health=11,kv=14,logm=48} crc {auth=2788676975,config=1228700967,health=797083738,kv=3374818887,logm=1832427657}) 2024-09-16T03:23:51.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.035+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=14,logm=46} crc {auth=752244947,config=1097430493,health=951176358,kv=3374818887,logm=3535489320}) 2024-09-16T03:23:51.038 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=12,logm=47} crc {auth=972967943,config=1097430493,health=866970265,kv=771521498,logm=3378136286}) 2024-09-16T03:23:51.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761180109}) 2024-09-16T03:23:51.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451648166}) 2024-09-16T03:23:51.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761180109}) 2024-09-16T03:23:51.041 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=111118290}) 2024-09-16T03:23:51.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 10 librados: watch_flush enter 2024-09-16T03:23:51.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 10 librados: watch_flush exit 2024-09-16T03:23:51.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 20 client.174225.objecter shutdown clearing up homeless session... 2024-09-16T03:23:51.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 10 client.174225.objecter successfully canceled tick 2024-09-16T03:23:51.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 >> v1:172.21.15.46:6812/149360280 conn(0x7f227c03e260 legacy=0x7f227c040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:51.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 10 monclient: shutdown 2024-09-16T03:23:51.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 >> v1:172.21.15.45:6789/0 conn(0x7f229813c010 legacy=0x7f22981d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 shutdown_connections 2024-09-16T03:23:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 >> 172.21.15.45:0/2992567689 conn(0x7f22980a2bd0 msgr2=0x7f22980a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 shutdown_connections 2024-09-16T03:23:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 1 -- 172.21.15.45:0/2992567689 wait complete. 2024-09-16T03:23:51.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:51.039+0000 7f22a0664640 1 librados: shutdown 2024-09-16T03:23:51.047 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.043+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.048 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.043+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=433036747}) 2024-09-16T03:23:51.048 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.043+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4137551248}) 2024-09-16T03:23:51.048 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.043+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.048 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.043+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=433036747}) 2024-09-16T03:23:51.048 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.043+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838054730}) 2024-09-16T03:23:51.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.047+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.047+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387678678}) 2024-09-16T03:23:51.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.047+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=821466339}) 2024-09-16T03:23:51.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.047+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.047+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387678678}) 2024-09-16T03:23:51.049 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.047+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1297492521}) 2024-09-16T03:23:51.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190236409}) 2024-09-16T03:23:51.051 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4288396694}) 2024-09-16T03:23:51.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190236409}) 2024-09-16T03:23:51.052 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2289723820}) 2024-09-16T03:23:51.054 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.054 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=677901766}) 2024-09-16T03:23:51.054 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156215960}) 2024-09-16T03:23:51.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=677901766}) 2024-09-16T03:23:51.055 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.051+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3409860912}) 2024-09-16T03:23:51.057 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:23:51.058 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.055+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.055+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149468432}) 2024-09-16T03:23:51.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.055+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1495790399}) 2024-09-16T03:23:51.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.055+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.055+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149468432}) 2024-09-16T03:23:51.059 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.055+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1697950678}) 2024-09-16T03:23:51.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.059+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.059+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299961414}) 2024-09-16T03:23:51.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.059+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3011940453}) 2024-09-16T03:23:51.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.059+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.059+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299961414}) 2024-09-16T03:23:51.061 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.059+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2206911856}) 2024-09-16T03:23:51.063 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.063+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.063 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.063+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350757538}) 2024-09-16T03:23:51.063 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.063+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3120016127}) 2024-09-16T03:23:51.063 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.063+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.063 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.063+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350757538}) 2024-09-16T03:23:51.063 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.063+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661512024}) 2024-09-16T03:23:51.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867230016}) 2024-09-16T03:23:51.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638083542}) 2024-09-16T03:23:51.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867230016}) 2024-09-16T03:23:51.066 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142191409}) 2024-09-16T03:23:51.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402551583}) 2024-09-16T03:23:51.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3938397107}) 2024-09-16T03:23:51.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402551583}) 2024-09-16T03:23:51.069 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.067+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448987625}) 2024-09-16T03:23:51.072 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.071+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.072 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.071+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002666218}) 2024-09-16T03:23:51.072 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.071+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1626483951}) 2024-09-16T03:23:51.072 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.071+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.072 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.071+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002666218}) 2024-09-16T03:23:51.072 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.071+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730227589}) 2024-09-16T03:23:51.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.075+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.075+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699527295}) 2024-09-16T03:23:51.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.075+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=794502557}) 2024-09-16T03:23:51.075 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.075+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.076 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.075+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699527295}) 2024-09-16T03:23:51.076 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.075+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=195677441}) 2024-09-16T03:23:51.078 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2507718703}) 2024-09-16T03:23:51.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3452735939}) 2024-09-16T03:23:51.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2507718703}) 2024-09-16T03:23:51.079 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3888731767}) 2024-09-16T03:23:51.081 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.081 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=47982019}) 2024-09-16T03:23:51.081 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1467021545}) 2024-09-16T03:23:51.082 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.082 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=47982019}) 2024-09-16T03:23:51.082 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.079+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2465438110}) 2024-09-16T03:23:51.084 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.083+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.084 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.083+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159783988}) 2024-09-16T03:23:51.084 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.083+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=931231811}) 2024-09-16T03:23:51.084 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.083+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.084 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.083+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159783988}) 2024-09-16T03:23:51.084 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.083+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2609194174}) 2024-09-16T03:23:51.087 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.087 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1091960024}) 2024-09-16T03:23:51.088 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=269275506}) 2024-09-16T03:23:51.088 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.088 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1091960024}) 2024-09-16T03:23:51.088 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3534755466}) 2024-09-16T03:23:51.090 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.090 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=339336175}) 2024-09-16T03:23:51.090 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3863935446}) 2024-09-16T03:23:51.090 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.090 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=339336175}) 2024-09-16T03:23:51.090 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.087+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3292626698}) 2024-09-16T03:23:51.092 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.091+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.092 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.091+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401611326}) 2024-09-16T03:23:51.092 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.091+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171849193}) 2024-09-16T03:23:51.093 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.091+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.093 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.091+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401611326}) 2024-09-16T03:23:51.093 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.091+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=754379761}) 2024-09-16T03:23:51.094 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.095 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767412954}) 2024-09-16T03:23:51.095 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4058447018}) 2024-09-16T03:23:51.095 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.095 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767412954}) 2024-09-16T03:23:51.095 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954467000}) 2024-09-16T03:23:51.097 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.097 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363336577}) 2024-09-16T03:23:51.097 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3528257992}) 2024-09-16T03:23:51.097 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.097 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363336577}) 2024-09-16T03:23:51.098 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.095+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182195239}) 2024-09-16T03:23:51.099 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.099 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311182570}) 2024-09-16T03:23:51.099 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=460196573}) 2024-09-16T03:23:51.100 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.100 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311182570}) 2024-09-16T03:23:51.100 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2094412777}) 2024-09-16T03:23:51.101 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.102 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840070815}) 2024-09-16T03:23:51.102 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4013757018}) 2024-09-16T03:23:51.102 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.102 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840070815}) 2024-09-16T03:23:51.102 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.099+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2967189647}) 2024-09-16T03:23:51.103 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.104 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595419573}) 2024-09-16T03:23:51.104 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2646836386}) 2024-09-16T03:23:51.104 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.104 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595419573}) 2024-09-16T03:23:51.104 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3854274187}) 2024-09-16T03:23:51.105 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.105 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217943830}) 2024-09-16T03:23:51.106 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=958450211}) 2024-09-16T03:23:51.106 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.106 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217943830}) 2024-09-16T03:23:51.106 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.103+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3682307916}) 2024-09-16T03:23:51.107 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3525700742}) 2024-09-16T03:23:51.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=528565562}) 2024-09-16T03:23:51.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3525700742}) 2024-09-16T03:23:51.108 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3533213383}) 2024-09-16T03:23:51.109 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.109 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456807243}) 2024-09-16T03:23:51.109 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1338251602}) 2024-09-16T03:23:51.110 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.110 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456807243}) 2024-09-16T03:23:51.110 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.107+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2047863162}) 2024-09-16T03:23:51.111 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.111 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413010818}) 2024-09-16T03:23:51.111 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=22091329}) 2024-09-16T03:23:51.111 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.112 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413010818}) 2024-09-16T03:23:51.112 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1746538853}) 2024-09-16T03:23:51.113 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.113 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531255836}) 2024-09-16T03:23:51.113 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3313944320}) 2024-09-16T03:23:51.113 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.113 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531255836}) 2024-09-16T03:23:51.114 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.111+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273572791}) 2024-09-16T03:23:51.115 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.115 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769411052}) 2024-09-16T03:23:51.116 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345639429}) 2024-09-16T03:23:51.116 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.116 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769411052}) 2024-09-16T03:23:51.116 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=866823997}) 2024-09-16T03:23:51.117 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.117 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190356482}) 2024-09-16T03:23:51.117 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2604913864}) 2024-09-16T03:23:51.117 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.118 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190356482}) 2024-09-16T03:23:51.118 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.115+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055173180}) 2024-09-16T03:23:51.119 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.119 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197209903}) 2024-09-16T03:23:51.119 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347107633}) 2024-09-16T03:23:51.119 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.119 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197209903}) 2024-09-16T03:23:51.120 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=60091369}) 2024-09-16T03:23:51.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414626783}) 2024-09-16T03:23:51.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=380279298}) 2024-09-16T03:23:51.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.121 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414626783}) 2024-09-16T03:23:51.122 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.119+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1349792688}) 2024-09-16T03:23:51.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940059182}) 2024-09-16T03:23:51.123 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3258297703}) 2024-09-16T03:23:51.124 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.124 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940059182}) 2024-09-16T03:23:51.124 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3923849008}) 2024-09-16T03:23:51.125 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.125 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337606471}) 2024-09-16T03:23:51.125 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442589534}) 2024-09-16T03:23:51.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337606471}) 2024-09-16T03:23:51.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=725599514}) 2024-09-16T03:23:51.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3974365767}) 2024-09-16T03:23:51.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1574258141}) 2024-09-16T03:23:51.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3974365767}) 2024-09-16T03:23:51.126 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.123+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2523752262}) 2024-09-16T03:23:51.127 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.127+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.127+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911165569}) 2024-09-16T03:23:51.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.127+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3739810350}) 2024-09-16T03:23:51.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.127+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.127+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911165569}) 2024-09-16T03:23:51.128 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.127+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4138126572}) 2024-09-16T03:23:51.130 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.130 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284920184}) 2024-09-16T03:23:51.130 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1306764025}) 2024-09-16T03:23:51.130 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.130 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284920184}) 2024-09-16T03:23:51.130 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20071554}) 2024-09-16T03:23:51.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779526503}) 2024-09-16T03:23:51.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417982585}) 2024-09-16T03:23:51.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779526503}) 2024-09-16T03:23:51.132 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.131+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912834626}) 2024-09-16T03:23:51.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964342335}) 2024-09-16T03:23:51.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2841967433}) 2024-09-16T03:23:51.134 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.135 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964342335}) 2024-09-16T03:23:51.135 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831862264}) 2024-09-16T03:23:51.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782052812}) 2024-09-16T03:23:51.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872365713}) 2024-09-16T03:23:51.136 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.137 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782052812}) 2024-09-16T03:23:51.137 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.135+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733384543}) 2024-09-16T03:23:51.138 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.138 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178257311}) 2024-09-16T03:23:51.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551298095}) 2024-09-16T03:23:51.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178257311}) 2024-09-16T03:23:51.139 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1412270156}) 2024-09-16T03:23:51.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=700863307}) 2024-09-16T03:23:51.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3106530884}) 2024-09-16T03:23:51.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=700863307}) 2024-09-16T03:23:51.141 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.139+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522247926}) 2024-09-16T03:23:51.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491009992}) 2024-09-16T03:23:51.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=54070129}) 2024-09-16T03:23:51.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491009992}) 2024-09-16T03:23:51.143 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3549312695}) 2024-09-16T03:23:51.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=542882386}) 2024-09-16T03:23:51.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2381153085}) 2024-09-16T03:23:51.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=542882386}) 2024-09-16T03:23:51.145 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.143+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3503325272}) 2024-09-16T03:23:51.148 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.148 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348514880}) 2024-09-16T03:23:51.148 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3715993438}) 2024-09-16T03:23:51.148 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348514880}) 2024-09-16T03:23:51.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=667612555}) 2024-09-16T03:23:51.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728411105}) 2024-09-16T03:23:51.149 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3745205936}) 2024-09-16T03:23:51.150 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.150 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728411105}) 2024-09-16T03:23:51.150 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.147+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=742213204}) 2024-09-16T03:23:51.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.151 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800179102}) 2024-09-16T03:23:51.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4113885860}) 2024-09-16T03:23:51.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800179102}) 2024-09-16T03:23:51.152 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=736526895}) 2024-09-16T03:23:51.153 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.153 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239032159}) 2024-09-16T03:23:51.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322926966}) 2024-09-16T03:23:51.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239032159}) 2024-09-16T03:23:51.154 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.151+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1354352769}) 2024-09-16T03:23:51.155 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.155+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.155+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2327476059}) 2024-09-16T03:23:51.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.155+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745776515}) 2024-09-16T03:23:51.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.155+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.155+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2327476059}) 2024-09-16T03:23:51.156 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.155+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1097293753}) 2024-09-16T03:23:51.158 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.155+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.158 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.155+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101146394}) 2024-09-16T03:23:51.158 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.155+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434251764}) 2024-09-16T03:23:51.158 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.158 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101146394}) 2024-09-16T03:23:51.158 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3500186553}) 2024-09-16T03:23:51.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258938203}) 2024-09-16T03:23:51.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4070911378}) 2024-09-16T03:23:51.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258938203}) 2024-09-16T03:23:51.160 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702360355}) 2024-09-16T03:23:51.161 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711905323}) 2024-09-16T03:23:51.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988423732}) 2024-09-16T03:23:51.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711905323}) 2024-09-16T03:23:51.162 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.159+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027295395}) 2024-09-16T03:23:51.163 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.163 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339754903}) 2024-09-16T03:23:51.163 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=189951041}) 2024-09-16T03:23:51.163 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339754903}) 2024-09-16T03:23:51.164 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1321365331}) 2024-09-16T03:23:51.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751201178}) 2024-09-16T03:23:51.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154176111}) 2024-09-16T03:23:51.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751201178}) 2024-09-16T03:23:51.165 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.163+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3475923550}) 2024-09-16T03:23:51.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=260353109}) 2024-09-16T03:23:51.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1929210749}) 2024-09-16T03:23:51.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=260353109}) 2024-09-16T03:23:51.167 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582092722}) 2024-09-16T03:23:51.168 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.169 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1169512339}) 2024-09-16T03:23:51.169 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937950846}) 2024-09-16T03:23:51.169 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.169 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1169512339}) 2024-09-16T03:23:51.169 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.167+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1685146609}) 2024-09-16T03:23:51.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432648688}) 2024-09-16T03:23:51.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676320083}) 2024-09-16T03:23:51.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432648688}) 2024-09-16T03:23:51.170 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1108148058}) 2024-09-16T03:23:51.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462399207}) 2024-09-16T03:23:51.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4162079614}) 2024-09-16T03:23:51.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462399207}) 2024-09-16T03:23:51.172 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2198404892}) 2024-09-16T03:23:51.173 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=963250826}) 2024-09-16T03:23:51.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3980199456}) 2024-09-16T03:23:51.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=963250826}) 2024-09-16T03:23:51.174 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.171+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916675168}) 2024-09-16T03:23:51.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702840930}) 2024-09-16T03:23:51.175 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=994359993}) 2024-09-16T03:23:51.176 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.176 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702840930}) 2024-09-16T03:23:51.176 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479709872}) 2024-09-16T03:23:51.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690525692}) 2024-09-16T03:23:51.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1213548824}) 2024-09-16T03:23:51.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690525692}) 2024-09-16T03:23:51.177 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.175+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256749005}) 2024-09-16T03:23:51.178 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952043938}) 2024-09-16T03:23:51.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=839077172}) 2024-09-16T03:23:51.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952043938}) 2024-09-16T03:23:51.179 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849490889}) 2024-09-16T03:23:51.180 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.180 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596089387}) 2024-09-16T03:23:51.180 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002114339}) 2024-09-16T03:23:51.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596089387}) 2024-09-16T03:23:51.181 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.179+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3483767339}) 2024-09-16T03:23:51.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813125674}) 2024-09-16T03:23:51.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4113750159}) 2024-09-16T03:23:51.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.182 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813125674}) 2024-09-16T03:23:51.183 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213849702}) 2024-09-16T03:23:51.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659635428}) 2024-09-16T03:23:51.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=195906740}) 2024-09-16T03:23:51.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659635428}) 2024-09-16T03:23:51.184 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=845505436}) 2024-09-16T03:23:51.185 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278516076}) 2024-09-16T03:23:51.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.183+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2418453939}) 2024-09-16T03:23:51.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278516076}) 2024-09-16T03:23:51.186 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237254529}) 2024-09-16T03:23:51.187 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.187 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846973627}) 2024-09-16T03:23:51.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1281894554}) 2024-09-16T03:23:51.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846973627}) 2024-09-16T03:23:51.188 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=823132222}) 2024-09-16T03:23:51.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3591267276}) 2024-09-16T03:23:51.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=25673069}) 2024-09-16T03:23:51.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.189 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3591267276}) 2024-09-16T03:23:51.190 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.187+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2531745394}) 2024-09-16T03:23:51.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1200512275}) 2024-09-16T03:23:51.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1208690484}) 2024-09-16T03:23:51.191 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.192 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1200512275}) 2024-09-16T03:23:51.192 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=282369945}) 2024-09-16T03:23:51.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431034238}) 2024-09-16T03:23:51.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2288154582}) 2024-09-16T03:23:51.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431034238}) 2024-09-16T03:23:51.193 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.191+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774874447}) 2024-09-16T03:23:51.194 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066259662}) 2024-09-16T03:23:51.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598381779}) 2024-09-16T03:23:51.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066259662}) 2024-09-16T03:23:51.195 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422539119}) 2024-09-16T03:23:51.196 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.196 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39820964}) 2024-09-16T03:23:51.196 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=9253467}) 2024-09-16T03:23:51.196 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39820964}) 2024-09-16T03:23:51.197 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.195+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701353637}) 2024-09-16T03:23:51.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451697003}) 2024-09-16T03:23:51.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1218962851}) 2024-09-16T03:23:51.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451697003}) 2024-09-16T03:23:51.198 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2672305971}) 2024-09-16T03:23:51.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.199 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033145898}) 2024-09-16T03:23:51.200 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933237396}) 2024-09-16T03:23:51.200 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.200 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033145898}) 2024-09-16T03:23:51.200 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975233040}) 2024-09-16T03:23:51.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084930808}) 2024-09-16T03:23:51.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1799858517}) 2024-09-16T03:23:51.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.201 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084930808}) 2024-09-16T03:23:51.202 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.199+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292324213}) 2024-09-16T03:23:51.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539175699}) 2024-09-16T03:23:51.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922359614}) 2024-09-16T03:23:51.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539175699}) 2024-09-16T03:23:51.203 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1207965255}) 2024-09-16T03:23:51.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=3715679493,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=3216643757,mon_config_key=2267004163,monmap=3372108098,nvmeofgw=1596038040,osd_metadata=2004720307,osd_pg_creating=4043048662,osd_snap=121161104}) 2024-09-16T03:23:51.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=19,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=1156262961,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=781948831,mon_config_key=1783149646,monmap=3140514009,nvmeofgw=598337686,osd_metadata=773870977,osd_pg_creating=4043048662,osd_snap=3403384283}) 2024-09-16T03:23:51.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=3715679493,mdsmap=2287870078,mgr=3873834454,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=3216643757,mon_config_key=2267004163,monmap=3372108098,nvmeofgw=1596038040,osd_metadata=2004720307,osd_pg_creating=4043048662,osd_snap=121161104}) 2024-09-16T03:23:51.206 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.203+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=31,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=17} crc {logm=4213089721,mdsmap=4056336170,mgr=3668741733,mgr_command_descs=2551042219,mgr_metadata=127159686,mgrstat=1678148100,mon_config_key=2843898858,monmap=3372108098,nvmeofgw=598337686,osd_metadata=2073814740,osd_pg_creating=4043048662,osd_snap=1243460795}) 2024-09-16T03:23:51.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.207+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.207+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=3003819717,osdmap=1720005796}) 2024-09-16T03:23:51.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.207+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=2876877207,osdmap=3685621228}) 2024-09-16T03:23:51.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.207+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.207+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=3003819717,osdmap=1720005796}) 2024-09-16T03:23:51.208 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.207+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=22,osdmap=78} crc {osd_snap=3362201834,osdmap=2197635756}) 2024-09-16T03:23:51.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.210 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3366409867}) 2024-09-16T03:23:51.211 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2467323851}) 2024-09-16T03:23:51.211 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.211 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3366409867}) 2024-09-16T03:23:51.211 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4126858417}) 2024-09-16T03:23:51.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=61268813}) 2024-09-16T03:23:51.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3378899139}) 2024-09-16T03:23:51.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=61268813}) 2024-09-16T03:23:51.212 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.211+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3333382212}) 2024-09-16T03:23:51.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2403394442}) 2024-09-16T03:23:51.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2689673487}) 2024-09-16T03:23:51.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2403394442}) 2024-09-16T03:23:51.214 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2269734163}) 2024-09-16T03:23:51.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3264839690}) 2024-09-16T03:23:51.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3091175297}) 2024-09-16T03:23:51.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3264839690}) 2024-09-16T03:23:51.216 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.215+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2463573634}) 2024-09-16T03:23:51.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.219+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.219+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2840173478}) 2024-09-16T03:23:51.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.219+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3945911293}) 2024-09-16T03:23:51.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.219+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.219+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2840173478}) 2024-09-16T03:23:51.218 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.219+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1998867360}) 2024-09-16T03:23:51.222 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.223+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.223+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3184587344}) 2024-09-16T03:23:51.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.223+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=183699632}) 2024-09-16T03:23:51.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.223+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.223+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3184587344}) 2024-09-16T03:23:51.223 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.223+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1347218833}) 2024-09-16T03:23:51.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.227+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.227+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1104887161}) 2024-09-16T03:23:51.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.227+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=959844573}) 2024-09-16T03:23:51.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.227+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.227+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1104887161}) 2024-09-16T03:23:51.227 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.227+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1401467249}) 2024-09-16T03:23:51.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.231+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.231+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2674982705}) 2024-09-16T03:23:51.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.231+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=664341072}) 2024-09-16T03:23:51.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.231+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:23:51.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.231+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2674982705}) 2024-09-16T03:23:51.231 INFO:tasks.ceph.mon.a.smithi045.stderr:2024-09-16T03:23:51.231+0000 7f727ad8e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2543154056}) 2024-09-16T03:23:52.059 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:23:52.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:23:52.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient: get_monmap_and_config 2024-09-16T03:23:52.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient: build_initial_monmap 2024-09-16T03:23:52.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient: monmap: 2024-09-16T03:23:52.191 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:52.191 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.192 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:52.193610+0000 2024-09-16T03:23:52.192 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:52.193610+0000 2024-09-16T03:23:52.192 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:52.192 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.192 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:52.192 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:52.192 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:52.192 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 1 Processor -- start 2024-09-16T03:23:52.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 1 -- start start 2024-09-16T03:23:52.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient: init 2024-09-16T03:23:52.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:52.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:52.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient(hunting): picked mon.noname-a con 0x7f07f409c570 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient(hunting): picked mon.noname-b con 0x7f07f4134320 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient(hunting): picked mon.noname-c con 0x7f07f41347d0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f4135150 con 0x7f07f409c570 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f4074470 con 0x7f07f41347d0 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f4074750 con 0x7f07f4134320 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 10 monclient(hunting): authenticate will time out at 2122.842952s 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f919f640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f07f4134320 0x7f07f413cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58406/0 (socket says 172.21.15.45:58406) 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f919f640 1 -- 172.21.15.45:0/1867182429 learned_addr learned my addr 172.21.15.45:0/1867182429 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:52.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f99a0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07f409c570 0x7f07f409c950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60226/0 (socket says 172.21.15.45:60226) 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2539562270 0 0) 0x7f07f0001120 con 0x7f07f409c570 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient(hunting): my global_id is 174228 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e0003530 con 0x7f07f409c570 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3212301059 0 0) 0x7f07f4074750 con 0x7f07f4134320 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient(hunting): my global_id is 184127 2024-09-16T03:23:52.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07f0001120 con 0x7f07f4134320 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1112995954 0 0) 0x7f07f0001120 con 0x7f07f4134320 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 >> v1:172.21.15.46:6789/0 conn(0x7f07f41347d0 legacy=0x7f07f41402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 >> v1:172.21.15.45:6789/0 conn(0x7f07f409c570 legacy=0x7f07f409c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient: _finish_hunting 0 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 monclient: found mon.noname-b 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f4074a30 con 0x7f07f4134320 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient: _finish_auth 0 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient: _check_auth_tickets 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f07e4001680 con 0x7f07f4134320 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 5 monclient: authenticate success, global_id 184127 2024-09-16T03:23:52.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07f899e640 10 monclient: dump: 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.191+0000 7f07fbc04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07e4001820 con 0x7f07f4134320 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07f899e640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 4 set_mon_vals no callback set 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07f899e640 1 -- 172.21.15.45:0/1867182429 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f07e4001bf0 con 0x7f07f4134320 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07f899e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07f899e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07f899e640 10 monclient: dump: 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: shutdown 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- 172.21.15.45:0/1867182429 >> v1:172.21.15.45:6790/0 conn(0x7f07f4134320 legacy=0x7f07f413cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- 172.21.15.45:0/1867182429 shutdown_connections 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- 172.21.15.45:0/1867182429 >> 172.21.15.45:0/1867182429 conn(0x7f07f40a2bd0 msgr2=0x7f07f40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- 172.21.15.45:0/1867182429 shutdown_connections 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- 172.21.15.45:0/1867182429 wait complete. 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: build_initial_monmap 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: monmap: 2024-09-16T03:23:52.205 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 librados: starting msgr at 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 librados: starting objecter 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 Processor -- start 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- start start 2024-09-16T03:23:52.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 librados: setting wanted keys 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 librados: calling monclient init 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: init 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient(hunting): picked mon.noname-b con 0x7f07f409c570 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient(hunting): picked mon.noname-a con 0x7f07f4134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient(hunting): picked mon.noname-c con 0x7f07f41347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f409d2c0 con 0x7f07f4134320 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f4133660 con 0x7f07f409c570 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f4133940 con 0x7f07f41347d0 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient(hunting): authenticate will time out at 2122.846002s 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fa1a1640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f07f41347d0 0x7f07f41d5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58422/0 (socket says 172.21.15.45:58422) 2024-09-16T03:23:52.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07f919f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07f4134320 0x7f07f41d1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60236/0 (socket says 172.21.15.45:60236) 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fa1a1640 1 -- 172.21.15.45:0/703983408 learned_addr learned my addr 172.21.15.45:0/703983408 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 client.?.objecter ms_handle_connect 0x7f07f4134320 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 client.?.objecter ms_handle_connect 0x7f07f41347d0 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3643856059 0 0) 0x7f07f4133940 con 0x7f07f41347d0 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): my global_id is 184130 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d8004ec0 con 0x7f07f41347d0 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839610348 0 0) 0x7f07f409d2c0 con 0x7f07f4134320 2024-09-16T03:23:52.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): my global_id is 174231 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07f4133940 con 0x7f07f4134320 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 client.?.objecter ms_handle_connect 0x7f07f409c570 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1228332775 0 0) 0x7f07d8004ec0 con 0x7f07f41347d0 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07f409d2c0 con 0x7f07f41347d0 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f07ec001d40 con 0x7f07f41347d0 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): dump: 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2742303881 0 0) 0x7f07f4133940 con 0x7f07f4134320 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07d8004ec0 con 0x7f07f4134320 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2088578819 0 0) 0x7f07f4133660 con 0x7f07f409c570 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): my global_id is 164311 2024-09-16T03:23:52.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07f4133940 con 0x7f07f409c570 2024-09-16T03:23:52.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f07e4001540 con 0x7f07f4134320 2024-09-16T03:23:52.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:52.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:52.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient(hunting): dump: 2024-09-16T03:23:52.211 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.211 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1177385904 0 0) 0x7f07f409d2c0 con 0x7f07f41347d0 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 >> v1:172.21.15.46:6789/0 conn(0x7f07f409c570 legacy=0x7f07f41cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 >> v1:172.21.15.45:6789/0 conn(0x7f07f4134320 legacy=0x7f07f41d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 monclient: found mon.c 2024-09-16T03:23:52.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f4133c20 con 0x7f07f41347d0 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient: _finish_auth 0 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 5 monclient: authenticate success, global_id 184130 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: _renew_subs 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07f4132700 con 0x7f07f41347d0 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 client.184130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: _renew_subs 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f07f4131230 con 0x7f07f41347d0 2024-09-16T03:23:52.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 librados: init done 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07ec0039a0 con 0x7f07f41347d0 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f07ec003d30 con 0x7f07f41347d0 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ebfff640 4 set_mon_vals no callback set 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 10 monclient: dump: 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.214 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f07ec003ed0 con 0x7f07f41347d0 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.195+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07dc005090 con 0x7f07f41347d0 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.199+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f07ec04b7a0 con 0x7f07f41347d0 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.199+0000 7f07ea7fc640 10 client.184130.objecter ms_dispatch 0x7f07f4000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.199+0000 7f07ea7fc640 3 client.184130.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.199+0000 7f07ea7fc640 3 client.184130.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.199+0000 7f07ea7fc640 20 client.184130.objecter dump_active .. 0 homeless 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.199+0000 7f07f99a0640 10 monclient: get_auth_request con 0x7f07d803e300 auth_method 0 2024-09-16T03:23:52.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.199+0000 7f07ea7fc640 10 client.184130.objecter ms_handle_connect 0x7f07d803e300 2024-09-16T03:23:52.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.203+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f07ec01b080 con 0x7f07f41347d0 2024-09-16T03:23:52.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.203+0000 7f07ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:52.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.203+0000 7f07ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:52.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.435+0000 7f07fbc04640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:23:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.435+0000 7f07fbc04640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:23:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.435+0000 7f07fbc04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.435+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f07dc005380 con 0x7f07f41347d0 2024-09-16T03:23:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.435+0000 7f07ea7fc640 1 -- 172.21.15.45:0/703983408 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1403 (unknown 254746657 0 576849531) 0x7f07ec01ffc0 con 0x7f07f41347d0 2024-09-16T03:23:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.435+0000 7f07ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:23:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.435+0000 7f07ea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:23:52.438 INFO:teuthology.orchestra.run.smithi045.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","modified":"2024-09-16T03:04:14.884162Z","created":"2024-09-16T03:04:14.884162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:23:52.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 10 librados: watch_flush enter 2024-09-16T03:23:52.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 10 librados: watch_flush exit 2024-09-16T03:23:52.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 20 client.184130.objecter shutdown clearing up homeless session... 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 10 client.184130.objecter successfully canceled tick 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 >> v1:172.21.15.46:6812/149360280 conn(0x7f07d803e300 legacy=0x7f07d80407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 10 monclient: shutdown 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 >> v1:172.21.15.45:6790/0 conn(0x7f07f41347d0 legacy=0x7f07f41d5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 shutdown_connections 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 >> 172.21.15.45:0/703983408 conn(0x7f07f40a2bd0 msgr2=0x7f07f409f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 shutdown_connections 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 1 -- 172.21.15.45:0/703983408 wait complete. 2024-09-16T03:23:52.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.439+0000 7f07fbc04640 1 librados: shutdown 2024-09-16T03:23:52.457 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:23:52.457 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T03:23:52.467 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:23:52.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:23:52.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.591+0000 7fc7b396a640 10 monclient: get_monmap_and_config 2024-09-16T03:23:52.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.591+0000 7fc7b396a640 10 monclient: build_initial_monmap 2024-09-16T03:23:52.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.591+0000 7fc7b396a640 10 monclient: monmap: 2024-09-16T03:23:52.593 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:52.593 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.593 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:52.595590+0000 2024-09-16T03:23:52.594 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:52.595590+0000 2024-09-16T03:23:52.594 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:52.594 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.594 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:52.594 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:52.594 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:52.594 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 Processor -- start 2024-09-16T03:23:52.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- start start 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient: init 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient(hunting): picked mon.noname-b con 0x7fc7ac137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient(hunting): picked mon.noname-c con 0x7fc7ac138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient(hunting): picked mon.noname-a con 0x7fc7ac13e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac0a4f90 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac0a4cb0 con 0x7fc7ac138100 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac09dc70 con 0x7fc7ac137d20 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient(hunting): authenticate will time out at 2123.245035s 2024-09-16T03:23:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b1f07640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc7ac13e2a0 0x7fc7ac140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60240/0 (socket says 172.21.15.45:60240) 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b1f07640 1 -- 172.21.15.45:0/1864504528 learned_addr learned my addr 172.21.15.45:0/1864504528 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3042986036 0 0) 0x7fc7ac0a4f90 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient(hunting): my global_id is 174234 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc798003530 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3690560263 0 0) 0x7fc7ac09dc70 con 0x7fc7ac137d20 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient(hunting): my global_id is 184133 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7ac0a4f90 con 0x7fc7ac137d20 2024-09-16T03:23:52.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 565870036 0 0) 0x7fc798003530 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 >> v1:172.21.15.45:6790/0 conn(0x7fc7ac137d20 legacy=0x7fc7ac0a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 >> v1:172.21.15.46:6789/0 conn(0x7fc7ac138100 legacy=0x7fc7ac13cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 monclient: found mon.noname-a 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac09cc50 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: _finish_auth 0 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 5 monclient: authenticate success, global_id 174234 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc7a4001a20 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:52.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: dump: 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7a4001c40 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 1 -- 172.21.15.45:0/1864504528 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc7a40020d0 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc79ffff640 10 monclient: dump: 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 4 set_mon_vals no callback set 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient: shutdown 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- 172.21.15.45:0/1864504528 >> v1:172.21.15.45:6789/0 conn(0x7fc7ac13e2a0 legacy=0x7fc7ac140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- 172.21.15.45:0/1864504528 shutdown_connections 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- 172.21.15.45:0/1864504528 >> 172.21.15.45:0/1864504528 conn(0x7fc7ac0aa110 msgr2=0x7fc7ac0aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- 172.21.15.45:0/1864504528 shutdown_connections 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- 172.21.15.45:0/1864504528 wait complete. 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient: build_initial_monmap 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient: monmap: 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:52.606 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 librados: starting msgr at 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 librados: starting objecter 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 Processor -- start 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 -- start start 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 librados: setting wanted keys 2024-09-16T03:23:52.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 1 librados: calling monclient init 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.595+0000 7fc7b396a640 10 monclient: init 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient(hunting): picked mon.noname-a con 0x7fc7ac137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient(hunting): picked mon.noname-b con 0x7fc7ac138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient(hunting): picked mon.noname-c con 0x7fc7ac13e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac13dbe0 con 0x7fc7ac137d20 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac138c30 con 0x7fc7ac138100 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac141aa0 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient(hunting): authenticate will time out at 2123.248365s 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b1706640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc7ac137d20 0x7fc7ac13cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60248/0 (socket says 172.21.15.45:60248) 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b1706640 1 -- 172.21.15.45:0/648281963 learned_addr learned my addr 172.21.15.45:0/648281963 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:52.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 client.?.objecter ms_handle_connect 0x7fc7ac137d20 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 client.?.objecter ms_handle_connect 0x7fc7ac13e2a0 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2804678225 0 0) 0x7fc7a8001140 con 0x7fc7ac137d20 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): my global_id is 174237 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc790004ec0 con 0x7fc7ac137d20 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1359137785 0 0) 0x7fc7ac141aa0 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): my global_id is 184136 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7a8001140 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 client.?.objecter ms_handle_connect 0x7fc7ac138100 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2087396049 0 0) 0x7fc7a8001140 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7ac141aa0 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 514584210 0 0) 0x7fc790004ec0 con 0x7fc7ac137d20 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7a8001140 con 0x7fc7ac137d20 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc7a4001df0 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): dump: 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.610 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc7a8001a10 con 0x7fc7ac137d20 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): dump: 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.611 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569550654 0 0) 0x7fc7ac138c30 con 0x7fc7ac138100 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient(hunting): my global_id is 164317 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc790004ec0 con 0x7fc7ac138100 2024-09-16T03:23:52.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2935010660 0 0) 0x7fc7ac141aa0 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 >> v1:172.21.15.46:6789/0 conn(0x7fc7ac138100 legacy=0x7fc7ac1d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 >> v1:172.21.15.45:6789/0 conn(0x7fc7ac137d20 legacy=0x7fc7ac13cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 monclient: found mon.c 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac0a3fe0 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient: _finish_auth 0 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 5 monclient: authenticate success, global_id 184136 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient: _renew_subs 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ac0a42c0 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 client.184136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient: _renew_subs 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc7ac141020 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 1 librados: init done 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7a4002090 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fc7a4005000 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79f7fe640 4 set_mon_vals no callback set 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 monclient: dump: 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.614 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc78c005090 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fc7a4011070 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7fc7a404b6f0 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 client.184136.objecter ms_dispatch 0x7fc7ac000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:52.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 3 client.184136.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:52.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 3 client.184136.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:52.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 20 client.184136.objecter dump_active .. 0 homeless 2024-09-16T03:23:52.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc7b1706640 10 monclient: get_auth_request con 0x7fc79003e290 auth_method 0 2024-09-16T03:23:52.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.599+0000 7fc79dffb640 10 client.184136.objecter ms_handle_connect 0x7fc79003e290 2024-09-16T03:23:52.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.603+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fc7a4016c90 con 0x7fc7ac13e2a0 2024-09-16T03:23:52.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.603+0000 7fc79dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:52.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.603+0000 7fc79dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:52.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.771+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc78c000ba0 con 0x7fc79003e290 2024-09-16T03:23:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.779+0000 7fc79dffb640 1 -- 172.21.15.45:0/648281963 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114047 (unknown 2965378022 0 1507276725) 0x7fc78c000ba0 con 0x7fc79003e290 2024-09-16T03:23:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:23:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-16T03:23:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.779+0000 7fc7b396a640 10 librados: watch_flush enter 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.779+0000 7fc7b396a640 10 librados: watch_flush exit 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 20 client.184136.objecter shutdown clearing up homeless session... 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 10 client.184136.objecter successfully canceled tick 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 >> v1:172.21.15.46:6812/149360280 conn(0x7fc79003e290 legacy=0x7fc790040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 10 monclient: shutdown 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 >> v1:172.21.15.45:6790/0 conn(0x7fc7ac13e2a0 legacy=0x7fc7ac1d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 shutdown_connections 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 >> 172.21.15.45:0/648281963 conn(0x7fc7ac0aa110 msgr2=0x7fc7ac130db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 shutdown_connections 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 1 -- 172.21.15.45:0/648281963 wait complete. 2024-09-16T03:23:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.783+0000 7fc7b396a640 1 librados: shutdown 2024-09-16T03:23:52.799 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1261,"stamp":"2024-09-16T03:23:51.870103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":469008,"kb_used_data":52776,"kb_used_omap":9,"kb_used_meta":416054,"kb_avail":565762032,"statfs":{"total":579820584960,"available":579340320768,"internally_reserved":0,"allocated":54042624,"data_stored":25919678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":426039970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004355"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210772+0000","last_change":"2024-09-16T03:06:02.756614+0000","last_active":"2024-09-16T03:22:27.210772+0000","last_peered":"2024-09-16T03:22:27.210772+0000","last_clean":"2024-09-16T03:22:27.210772+0000","last_became_active":"2024-09-16T03:06:02.756187+0000","last_became_peered":"2024-09-16T03:06:02.756187+0000","last_unstale":"2024-09-16T03:22:27.210772+0000","last_undegraded":"2024-09-16T03:22:27.210772+0000","last_fullsized":"2024-09-16T03:22:27.210772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:43.602727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211989+0000","last_change":"2024-09-16T03:06:02.806920+0000","last_active":"2024-09-16T03:22:27.211989+0000","last_peered":"2024-09-16T03:22:27.211989+0000","last_clean":"2024-09-16T03:22:27.211989+0000","last_became_active":"2024-09-16T03:06:02.798283+0000","last_became_peered":"2024-09-16T03:06:02.798283+0000","last_unstale":"2024-09-16T03:22:27.211989+0000","last_undegraded":"2024-09-16T03:22:27.211989+0000","last_fullsized":"2024-09-16T03:22:27.211989+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:43.018291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.212003+0000","last_change":"2024-09-16T03:06:02.832611+0000","last_active":"2024-09-16T03:22:27.212003+0000","last_peered":"2024-09-16T03:22:27.212003+0000","last_clean":"2024-09-16T03:22:27.212003+0000","last_became_active":"2024-09-16T03:06:02.830437+0000","last_became_peered":"2024-09-16T03:06:02.830437+0000","last_unstale":"2024-09-16T03:22:27.212003+0000","last_undegraded":"2024-09-16T03:22:27.212003+0000","last_fullsized":"2024-09-16T03:22:27.212003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:25.137935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774614+0000","last_change":"2024-09-16T03:06:02.664001+0000","last_active":"2024-09-16T03:22:26.774614+0000","last_peered":"2024-09-16T03:22:26.774614+0000","last_clean":"2024-09-16T03:22:26.774614+0000","last_became_active":"2024-09-16T03:06:02.661502+0000","last_became_peered":"2024-09-16T03:06:02.661502+0000","last_unstale":"2024-09-16T03:22:26.774614+0000","last_undegraded":"2024-09-16T03:22:26.774614+0000","last_fullsized":"2024-09-16T03:22:26.774614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:56.502316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209178+0000","last_change":"2024-09-16T03:06:02.750362+0000","last_active":"2024-09-16T03:22:27.209178+0000","last_peered":"2024-09-16T03:22:27.209178+0000","last_clean":"2024-09-16T03:22:27.209178+0000","last_became_active":"2024-09-16T03:06:02.744921+0000","last_became_peered":"2024-09-16T03:06:02.744921+0000","last_unstale":"2024-09-16T03:22:27.209178+0000","last_undegraded":"2024-09-16T03:22:27.209178+0000","last_fullsized":"2024-09-16T03:22:27.209178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:03.960933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1336,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777944+0000","last_change":"2024-09-16T03:06:02.747722+0000","last_active":"2024-09-16T03:22:26.777944+0000","last_peered":"2024-09-16T03:22:26.777944+0000","last_clean":"2024-09-16T03:22:26.777944+0000","last_became_active":"2024-09-16T03:06:02.746529+0000","last_became_peered":"2024-09-16T03:06:02.746529+0000","last_unstale":"2024-09-16T03:22:26.777944+0000","last_undegraded":"2024-09-16T03:22:26.777944+0000","last_fullsized":"2024-09-16T03:22:26.777944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:54.961886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209716+0000","last_change":"2024-09-16T03:06:02.755623+0000","last_active":"2024-09-16T03:22:27.209716+0000","last_peered":"2024-09-16T03:22:27.209716+0000","last_clean":"2024-09-16T03:22:27.209716+0000","last_became_active":"2024-09-16T03:06:02.755275+0000","last_became_peered":"2024-09-16T03:06:02.755275+0000","last_unstale":"2024-09-16T03:22:27.209716+0000","last_undegraded":"2024-09-16T03:22:27.209716+0000","last_fullsized":"2024-09-16T03:22:27.209716+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:53.461709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122045+0000","last_change":"2024-09-16T03:06:02.820568+0000","last_active":"2024-09-16T03:22:27.122045+0000","last_peered":"2024-09-16T03:22:27.122045+0000","last_clean":"2024-09-16T03:22:27.122045+0000","last_became_active":"2024-09-16T03:06:02.820265+0000","last_became_peered":"2024-09-16T03:06:02.820265+0000","last_unstale":"2024-09-16T03:22:27.122045+0000","last_undegraded":"2024-09-16T03:22:27.122045+0000","last_fullsized":"2024-09-16T03:22:27.122045+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:30.846375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774889+0000","last_change":"2024-09-16T03:06:02.699671+0000","last_active":"2024-09-16T03:22:26.774889+0000","last_peered":"2024-09-16T03:22:26.774889+0000","last_clean":"2024-09-16T03:22:26.774889+0000","last_became_active":"2024-09-16T03:06:02.698531+0000","last_became_peered":"2024-09-16T03:06:02.698531+0000","last_unstale":"2024-09-16T03:22:26.774889+0000","last_undegraded":"2024-09-16T03:22:26.774889+0000","last_fullsized":"2024-09-16T03:22:26.774889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:19:31.192437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777641+0000","last_change":"2024-09-16T03:06:02.829600+0000","last_active":"2024-09-16T03:22:26.777641+0000","last_peered":"2024-09-16T03:22:26.777641+0000","last_clean":"2024-09-16T03:22:26.777641+0000","last_became_active":"2024-09-16T03:06:02.829225+0000","last_became_peered":"2024-09-16T03:06:02.829225+0000","last_unstale":"2024-09-16T03:22:26.777641+0000","last_undegraded":"2024-09-16T03:22:26.777641+0000","last_fullsized":"2024-09-16T03:22:26.777641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:02.011377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211564+0000","last_change":"2024-09-16T03:06:02.755653+0000","last_active":"2024-09-16T03:22:27.211564+0000","last_peered":"2024-09-16T03:22:27.211564+0000","last_clean":"2024-09-16T03:22:27.211564+0000","last_became_active":"2024-09-16T03:06:02.750016+0000","last_became_peered":"2024-09-16T03:06:02.750016+0000","last_unstale":"2024-09-16T03:22:27.211564+0000","last_undegraded":"2024-09-16T03:22:27.211564+0000","last_fullsized":"2024-09-16T03:22:27.211564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:44.526283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777284+0000","last_change":"2024-09-16T03:06:02.858185+0000","last_active":"2024-09-16T03:22:26.777284+0000","last_peered":"2024-09-16T03:22:26.777284+0000","last_clean":"2024-09-16T03:22:26.777284+0000","last_became_active":"2024-09-16T03:06:02.857164+0000","last_became_peered":"2024-09-16T03:06:02.857164+0000","last_unstale":"2024-09-16T03:22:26.777284+0000","last_undegraded":"2024-09-16T03:22:26.777284+0000","last_fullsized":"2024-09-16T03:22:26.777284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:44.538058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1356,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209549+0000","last_change":"2024-09-16T03:04:59.132718+0000","last_active":"2024-09-16T03:22:27.209549+0000","last_peered":"2024-09-16T03:22:27.209549+0000","last_clean":"2024-09-16T03:22:27.209549+0000","last_became_active":"2024-09-16T03:04:57.691813+0000","last_became_peered":"2024-09-16T03:04:57.691813+0000","last_unstale":"2024-09-16T03:22:27.209549+0000","last_undegraded":"2024-09-16T03:22:27.209549+0000","last_fullsized":"2024-09-16T03:22:27.209549+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:55.115695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015712199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210838+0000","last_change":"2024-09-16T03:06:02.721325+0000","last_active":"2024-09-16T03:22:27.210838+0000","last_peered":"2024-09-16T03:22:27.210838+0000","last_clean":"2024-09-16T03:22:27.210838+0000","last_became_active":"2024-09-16T03:06:02.721049+0000","last_became_peered":"2024-09-16T03:06:02.721049+0000","last_unstale":"2024-09-16T03:22:27.210838+0000","last_undegraded":"2024-09-16T03:22:27.210838+0000","last_fullsized":"2024-09-16T03:22:27.210838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:56.615690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1363,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.123007+0000","last_change":"2024-09-16T03:04:59.142318+0000","last_active":"2024-09-16T03:22:27.123007+0000","last_peered":"2024-09-16T03:22:27.123007+0000","last_clean":"2024-09-16T03:22:27.123007+0000","last_became_active":"2024-09-16T03:04:57.117464+0000","last_became_peered":"2024-09-16T03:04:57.117464+0000","last_unstale":"2024-09-16T03:22:27.123007+0000","last_undegraded":"2024-09-16T03:22:27.123007+0000","last_fullsized":"2024-09-16T03:22:27.123007+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:54:40.258398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012748499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209678+0000","last_change":"2024-09-16T03:06:02.769149+0000","last_active":"2024-09-16T03:22:27.209678+0000","last_peered":"2024-09-16T03:22:27.209678+0000","last_clean":"2024-09-16T03:22:27.209678+0000","last_became_active":"2024-09-16T03:06:02.768926+0000","last_became_peered":"2024-09-16T03:06:02.768926+0000","last_unstale":"2024-09-16T03:22:27.209678+0000","last_undegraded":"2024-09-16T03:22:27.209678+0000","last_fullsized":"2024-09-16T03:22:27.209678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:11.356942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1357,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209966+0000","last_change":"2024-09-16T03:04:59.131904+0000","last_active":"2024-09-16T03:22:27.209966+0000","last_peered":"2024-09-16T03:22:27.209966+0000","last_clean":"2024-09-16T03:22:27.209966+0000","last_became_active":"2024-09-16T03:04:57.113180+0000","last_became_peered":"2024-09-16T03:04:57.113180+0000","last_unstale":"2024-09-16T03:22:27.209966+0000","last_undegraded":"2024-09-16T03:22:27.209966+0000","last_fullsized":"2024-09-16T03:22:27.209966+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:30.278408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087237100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1361,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122484+0000","last_change":"2024-09-16T03:06:02.678570+0000","last_active":"2024-09-16T03:22:27.122484+0000","last_peered":"2024-09-16T03:22:27.122484+0000","last_clean":"2024-09-16T03:22:27.122484+0000","last_became_active":"2024-09-16T03:06:02.662210+0000","last_became_peered":"2024-09-16T03:06:02.662210+0000","last_unstale":"2024-09-16T03:22:27.122484+0000","last_undegraded":"2024-09-16T03:22:27.122484+0000","last_fullsized":"2024-09-16T03:22:27.122484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:18.499817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1363,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122893+0000","last_change":"2024-09-16T03:04:59.142273+0000","last_active":"2024-09-16T03:22:27.122893+0000","last_peered":"2024-09-16T03:22:27.122893+0000","last_clean":"2024-09-16T03:22:27.122893+0000","last_became_active":"2024-09-16T03:04:57.117660+0000","last_became_peered":"2024-09-16T03:04:57.117660+0000","last_unstale":"2024-09-16T03:22:27.122893+0000","last_undegraded":"2024-09-16T03:22:27.122893+0000","last_fullsized":"2024-09-16T03:22:27.122893+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:22.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211578+0000","last_change":"2024-09-16T03:06:02.853549+0000","last_active":"2024-09-16T03:22:27.211578+0000","last_peered":"2024-09-16T03:22:27.211578+0000","last_clean":"2024-09-16T03:22:27.211578+0000","last_became_active":"2024-09-16T03:06:02.852993+0000","last_became_peered":"2024-09-16T03:06:02.852993+0000","last_unstale":"2024-09-16T03:22:27.211578+0000","last_undegraded":"2024-09-16T03:22:27.211578+0000","last_fullsized":"2024-09-16T03:22:27.211578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:42.416142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774729+0000","last_change":"2024-09-16T03:04:59.138496+0000","last_active":"2024-09-16T03:22:26.774729+0000","last_peered":"2024-09-16T03:22:26.774729+0000","last_clean":"2024-09-16T03:22:26.774729+0000","last_became_active":"2024-09-16T03:04:57.119695+0000","last_became_peered":"2024-09-16T03:04:57.119695+0000","last_unstale":"2024-09-16T03:22:26.774729+0000","last_undegraded":"2024-09-16T03:22:26.774729+0000","last_fullsized":"2024-09-16T03:22:26.774729+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:24.631318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024209740000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.123006+0000","last_change":"2024-09-16T03:06:02.765932+0000","last_active":"2024-09-16T03:22:27.123006+0000","last_peered":"2024-09-16T03:22:27.123006+0000","last_clean":"2024-09-16T03:22:27.123006+0000","last_became_active":"2024-09-16T03:06:02.760889+0000","last_became_peered":"2024-09-16T03:06:02.760889+0000","last_unstale":"2024-09-16T03:22:27.123006+0000","last_undegraded":"2024-09-16T03:22:27.123006+0000","last_fullsized":"2024-09-16T03:22:27.123006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:26.594180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1352,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777443+0000","last_change":"2024-09-16T03:04:59.133304+0000","last_active":"2024-09-16T03:22:26.777443+0000","last_peered":"2024-09-16T03:22:26.777443+0000","last_clean":"2024-09-16T03:22:26.777443+0000","last_became_active":"2024-09-16T03:04:57.118413+0000","last_became_peered":"2024-09-16T03:04:57.118413+0000","last_unstale":"2024-09-16T03:22:26.777443+0000","last_undegraded":"2024-09-16T03:22:26.777443+0000","last_fullsized":"2024-09-16T03:22:26.777443+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:24.300841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014587720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774981+0000","last_change":"2024-09-16T03:06:02.746071+0000","last_active":"2024-09-16T03:22:26.774981+0000","last_peered":"2024-09-16T03:22:26.774981+0000","last_clean":"2024-09-16T03:22:26.774981+0000","last_became_active":"2024-09-16T03:06:02.743279+0000","last_became_peered":"2024-09-16T03:06:02.743279+0000","last_unstale":"2024-09-16T03:22:26.774981+0000","last_undegraded":"2024-09-16T03:22:26.774981+0000","last_fullsized":"2024-09-16T03:22:26.774981+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:12.479265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1357,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209320+0000","last_change":"2024-09-16T03:04:59.132034+0000","last_active":"2024-09-16T03:22:27.209320+0000","last_peered":"2024-09-16T03:22:27.209320+0000","last_clean":"2024-09-16T03:22:27.209320+0000","last_became_active":"2024-09-16T03:04:57.117048+0000","last_became_peered":"2024-09-16T03:04:57.117048+0000","last_unstale":"2024-09-16T03:22:27.209320+0000","last_undegraded":"2024-09-16T03:22:27.209320+0000","last_fullsized":"2024-09-16T03:22:27.209320+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:57.899748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099232000000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122142+0000","last_change":"2024-09-16T03:06:02.822684+0000","last_active":"2024-09-16T03:22:27.122142+0000","last_peered":"2024-09-16T03:22:27.122142+0000","last_clean":"2024-09-16T03:22:27.122142+0000","last_became_active":"2024-09-16T03:06:02.820554+0000","last_became_peered":"2024-09-16T03:06:02.820554+0000","last_unstale":"2024-09-16T03:22:27.122142+0000","last_undegraded":"2024-09-16T03:22:27.122142+0000","last_fullsized":"2024-09-16T03:22:27.122142+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:24:48.622458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774841+0000","last_change":"2024-09-16T03:04:59.137133+0000","last_active":"2024-09-16T03:22:26.774841+0000","last_peered":"2024-09-16T03:22:26.774841+0000","last_clean":"2024-09-16T03:22:26.774841+0000","last_became_active":"2024-09-16T03:04:57.116849+0000","last_became_peered":"2024-09-16T03:04:57.116849+0000","last_unstale":"2024-09-16T03:22:26.774841+0000","last_undegraded":"2024-09-16T03:22:26.774841+0000","last_fullsized":"2024-09-16T03:22:26.774841+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:51.385856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00103192,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1448,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209486+0000","last_change":"2024-09-16T03:04:51.319467+0000","last_active":"2024-09-16T03:22:27.209486+0000","last_peered":"2024-09-16T03:22:27.209486+0000","last_clean":"2024-09-16T03:22:27.209486+0000","last_became_active":"2024-09-16T03:04:51.317160+0000","last_became_peered":"2024-09-16T03:04:51.317160+0000","last_unstale":"2024-09-16T03:22:27.209486+0000","last_undegraded":"2024-09-16T03:22:27.209486+0000","last_fullsized":"2024-09-16T03:22:27.209486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_clean_scrub_stamp":"2024-09-16T03:04:49.540253+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:15.016218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.775063+0000","last_change":"2024-09-16T03:06:02.762194+0000","last_active":"2024-09-16T03:22:26.775063+0000","last_peered":"2024-09-16T03:22:26.775063+0000","last_clean":"2024-09-16T03:22:26.775063+0000","last_became_active":"2024-09-16T03:06:02.761928+0000","last_became_peered":"2024-09-16T03:06:02.761928+0000","last_unstale":"2024-09-16T03:22:26.775063+0000","last_undegraded":"2024-09-16T03:22:26.775063+0000","last_fullsized":"2024-09-16T03:22:26.775063+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:27:32.863289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774795+0000","last_change":"2024-09-16T03:06:02.664328+0000","last_active":"2024-09-16T03:22:26.774795+0000","last_peered":"2024-09-16T03:22:26.774795+0000","last_clean":"2024-09-16T03:22:26.774795+0000","last_became_active":"2024-09-16T03:06:02.661047+0000","last_became_peered":"2024-09-16T03:06:02.661047+0000","last_unstale":"2024-09-16T03:22:26.774795+0000","last_undegraded":"2024-09-16T03:22:26.774795+0000","last_fullsized":"2024-09-16T03:22:26.774795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:52.055646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1346,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122655+0000","last_change":"2024-09-16T03:06:02.769535+0000","last_active":"2024-09-16T03:22:27.122655+0000","last_peered":"2024-09-16T03:22:27.122655+0000","last_clean":"2024-09-16T03:22:27.122655+0000","last_became_active":"2024-09-16T03:06:02.769042+0000","last_became_peered":"2024-09-16T03:06:02.769042+0000","last_unstale":"2024-09-16T03:22:27.122655+0000","last_undegraded":"2024-09-16T03:22:27.122655+0000","last_fullsized":"2024-09-16T03:22:27.122655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:26.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777928+0000","last_change":"2024-09-16T03:06:02.829591+0000","last_active":"2024-09-16T03:22:26.777928+0000","last_peered":"2024-09-16T03:22:26.777928+0000","last_clean":"2024-09-16T03:22:26.777928+0000","last_became_active":"2024-09-16T03:06:02.829311+0000","last_became_peered":"2024-09-16T03:06:02.829311+0000","last_unstale":"2024-09-16T03:22:26.777928+0000","last_undegraded":"2024-09-16T03:22:26.777928+0000","last_fullsized":"2024-09-16T03:22:26.777928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:04.717263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211030+0000","last_change":"2024-09-16T03:06:02.821360+0000","last_active":"2024-09-16T03:22:27.211030+0000","last_peered":"2024-09-16T03:22:27.211030+0000","last_clean":"2024-09-16T03:22:27.211030+0000","last_became_active":"2024-09-16T03:06:02.820637+0000","last_became_peered":"2024-09-16T03:06:02.820637+0000","last_unstale":"2024-09-16T03:22:27.211030+0000","last_undegraded":"2024-09-16T03:22:27.211030+0000","last_fullsized":"2024-09-16T03:22:27.211030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:56.743641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211911+0000","last_change":"2024-09-16T03:06:02.800273+0000","last_active":"2024-09-16T03:22:27.211911+0000","last_peered":"2024-09-16T03:22:27.211911+0000","last_clean":"2024-09-16T03:22:27.211911+0000","last_became_active":"2024-09-16T03:06:02.798149+0000","last_became_peered":"2024-09-16T03:06:02.798149+0000","last_unstale":"2024-09-16T03:22:27.211911+0000","last_undegraded":"2024-09-16T03:22:27.211911+0000","last_fullsized":"2024-09-16T03:22:27.211911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:44.563168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209139+0000","last_change":"2024-09-16T03:06:02.739124+0000","last_active":"2024-09-16T03:22:27.209139+0000","last_peered":"2024-09-16T03:22:27.209139+0000","last_clean":"2024-09-16T03:22:27.209139+0000","last_became_active":"2024-09-16T03:06:02.736697+0000","last_became_peered":"2024-09-16T03:06:02.736697+0000","last_unstale":"2024-09-16T03:22:27.209139+0000","last_undegraded":"2024-09-16T03:22:27.209139+0000","last_fullsized":"2024-09-16T03:22:27.209139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:56:28.860331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122618+0000","last_change":"2024-09-16T03:06:02.661500+0000","last_active":"2024-09-16T03:22:27.122618+0000","last_peered":"2024-09-16T03:22:27.122618+0000","last_clean":"2024-09-16T03:22:27.122618+0000","last_became_active":"2024-09-16T03:06:02.660659+0000","last_became_peered":"2024-09-16T03:06:02.660659+0000","last_unstale":"2024-09-16T03:22:27.122618+0000","last_undegraded":"2024-09-16T03:22:27.122618+0000","last_fullsized":"2024-09-16T03:22:27.122618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:50.530990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774695+0000","last_change":"2024-09-16T03:06:02.675225+0000","last_active":"2024-09-16T03:22:26.774695+0000","last_peered":"2024-09-16T03:22:26.774695+0000","last_clean":"2024-09-16T03:22:26.774695+0000","last_became_active":"2024-09-16T03:06:02.668156+0000","last_became_peered":"2024-09-16T03:06:02.668156+0000","last_unstale":"2024-09-16T03:22:26.774695+0000","last_undegraded":"2024-09-16T03:22:26.774695+0000","last_fullsized":"2024-09-16T03:22:26.774695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:10:19.961091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209688+0000","last_change":"2024-09-16T03:06:02.755239+0000","last_active":"2024-09-16T03:22:27.209688+0000","last_peered":"2024-09-16T03:22:27.209688+0000","last_clean":"2024-09-16T03:22:27.209688+0000","last_became_active":"2024-09-16T03:06:02.752964+0000","last_became_peered":"2024-09-16T03:06:02.752964+0000","last_unstale":"2024-09-16T03:22:27.209688+0000","last_undegraded":"2024-09-16T03:22:27.209688+0000","last_fullsized":"2024-09-16T03:22:27.209688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:17.029098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777476+0000","last_change":"2024-09-16T03:06:02.749131+0000","last_active":"2024-09-16T03:22:26.777476+0000","last_peered":"2024-09-16T03:22:26.777476+0000","last_clean":"2024-09-16T03:22:26.777476+0000","last_became_active":"2024-09-16T03:06:02.748671+0000","last_became_peered":"2024-09-16T03:06:02.748671+0000","last_unstale":"2024-09-16T03:22:26.777476+0000","last_undegraded":"2024-09-16T03:22:26.777476+0000","last_fullsized":"2024-09-16T03:22:26.777476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.998002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210912+0000","last_change":"2024-09-16T03:06:02.723416+0000","last_active":"2024-09-16T03:22:27.210912+0000","last_peered":"2024-09-16T03:22:27.210912+0000","last_clean":"2024-09-16T03:22:27.210912+0000","last_became_active":"2024-09-16T03:06:02.723112+0000","last_became_peered":"2024-09-16T03:06:02.723112+0000","last_unstale":"2024-09-16T03:22:27.210912+0000","last_undegraded":"2024-09-16T03:22:27.210912+0000","last_fullsized":"2024-09-16T03:22:27.210912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:02.720767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.212250+0000","last_change":"2024-09-16T03:06:02.845346+0000","last_active":"2024-09-16T03:22:27.212250+0000","last_peered":"2024-09-16T03:22:27.212250+0000","last_clean":"2024-09-16T03:22:27.212250+0000","last_became_active":"2024-09-16T03:06:02.843866+0000","last_became_peered":"2024-09-16T03:06:02.843866+0000","last_unstale":"2024-09-16T03:22:27.212250+0000","last_undegraded":"2024-09-16T03:22:27.212250+0000","last_fullsized":"2024-09-16T03:22:27.212250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:45.180460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607782,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84240,"kb_used_data":8624,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287600,"statfs":{"total":96636764160,"available":96550502400,"internally_reserved":0,"allocated":8830976,"data_stored":4143648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607783,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71748,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300092,"statfs":{"total":96636764160,"available":96563294208,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607782,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84752,"kb_used_data":9144,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287088,"statfs":{"total":96636764160,"available":96549978112,"internally_reserved":0,"allocated":9363456,"data_stored":4672562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607782,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71764,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300076,"statfs":{"total":96636764160,"available":96563277824,"internally_reserved":0,"allocated":8835072,"data_stored":4143653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640487,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72276,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94299564,"statfs":{"total":96636764160,"available":96562753536,"internally_reserved":0,"allocated":9359360,"data_stored":4672557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84228,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287612,"statfs":{"total":96636764160,"available":96550514688,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:23:52.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:23:52.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.923+0000 7fdc7cf25640 10 monclient: get_monmap_and_config 2024-09-16T03:23:52.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.923+0000 7fdc7cf25640 10 monclient: build_initial_monmap 2024-09-16T03:23:52.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.923+0000 7fdc7cf25640 10 monclient: monmap: 2024-09-16T03:23:52.925 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:52.926 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.926 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:52.927752+0000 2024-09-16T03:23:52.926 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:52.927752+0000 2024-09-16T03:23:52.926 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:52.926 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.926 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:52.926 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:52.926 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:52.926 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 Processor -- start 2024-09-16T03:23:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- start start 2024-09-16T03:23:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient: init 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient(hunting): picked mon.noname-b con 0x7fdc78135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7813c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7809f7e0 con 0x7fdc7813c010 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7809b160 con 0x7fdc7809fc40 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7809b440 con 0x7fdc78135950 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient(hunting): authenticate will time out at 2123.577362s 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdc78135950 0x7fdc78135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58434/0 (socket says 172.21.15.45:58434) 2024-09-16T03:23:52.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7659c640 1 -- 172.21.15.45:0/2987558904 learned_addr learned my addr 172.21.15.45:0/2987558904 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc76d9d640 1 --1- 172.21.15.45:0/2987558904 >> v1:172.21.15.45:6789/0 conn(0x7fdc7813c010 0x7fdc7813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60250/0 (socket says 172.21.15.45:60250) 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1690026365 0 0) 0x7fdc7809f7e0 con 0x7fdc7813c010 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient(hunting): my global_id is 174240 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc60003530 con 0x7fdc7813c010 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1444646316 0 0) 0x7fdc7809b440 con 0x7fdc78135950 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient(hunting): my global_id is 184139 2024-09-16T03:23:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7809f7e0 con 0x7fdc78135950 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2153594497 0 0) 0x7fdc60003530 con 0x7fdc7813c010 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 >> v1:172.21.15.45:6790/0 conn(0x7fdc78135950 legacy=0x7fdc78135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 >> v1:172.21.15.46:6789/0 conn(0x7fdc7809fc40 legacy=0x7fdc7813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: _finish_hunting 0 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 monclient: found mon.noname-a 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc78136440 con 0x7fdc7813c010 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: _finish_auth 0 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: _check_auth_tickets 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdc70001a20 con 0x7fdc7813c010 2024-09-16T03:23:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: dump: 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 5 monclient: authenticate success, global_id 174240 2024-09-16T03:23:52.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc70001bc0 con 0x7fdc7813c010 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 1 -- 172.21.15.45:0/2987558904 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdc70002050 con 0x7fdc7813c010 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 4 set_mon_vals no callback set 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7559a640 10 monclient: dump: 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient: shutdown 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2987558904 >> v1:172.21.15.45:6789/0 conn(0x7fdc7813c010 legacy=0x7fdc7813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2987558904 shutdown_connections 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2987558904 >> 172.21.15.45:0/2987558904 conn(0x7fdc780a2bd0 msgr2=0x7fdc780a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2987558904 shutdown_connections 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2987558904 wait complete. 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient: build_initial_monmap 2024-09-16T03:23:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient: monmap: 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 librados: starting msgr at 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 librados: starting objecter 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 Processor -- start 2024-09-16T03:23:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 -- start start 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 librados: setting wanted keys 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 1 librados: calling monclient init 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.927+0000 7fdc7cf25640 10 monclient: init 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7809fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient(hunting): picked mon.noname-c con 0x7fdc78135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient(hunting): picked mon.noname-b con 0x7fdc7813c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc780718e0 con 0x7fdc7809fc40 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7813bb90 con 0x7fdc7813c010 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7813b6f0 con 0x7fdc78135950 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient(hunting): authenticate will time out at 2123.580315s 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdc7809fc40 0x7fdc7806e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60254/0 (socket says 172.21.15.45:60254) 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc75d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdc78135950 0x7fdc7806ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58446/0 (socket says 172.21.15.45:58446) 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7659c640 1 -- 172.21.15.45:0/2552751971 learned_addr learned my addr 172.21.15.45:0/2552751971 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 client.?.objecter ms_handle_connect 0x7fdc7809fc40 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 client.?.objecter ms_handle_connect 0x7fdc78135950 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728654162 0 0) 0x7fdc7813b6f0 con 0x7fdc78135950 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): my global_id is 184142 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc58004ef0 con 0x7fdc78135950 2024-09-16T03:23:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530628553 0 0) 0x7fdc780718e0 con 0x7fdc7809fc40 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): my global_id is 174243 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7813b6f0 con 0x7fdc7809fc40 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 client.?.objecter ms_handle_connect 0x7fdc7813c010 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1961485039 0 0) 0x7fdc58004ef0 con 0x7fdc78135950 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc780718e0 con 0x7fdc78135950 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdc68001860 con 0x7fdc78135950 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): dump: 2024-09-16T03:23:52.944 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2476142633 0 0) 0x7fdc7813b6f0 con 0x7fdc7809fc40 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc58004ef0 con 0x7fdc7809fc40 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdc6c0019d0 con 0x7fdc7809fc40 2024-09-16T03:23:52.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): dump: 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366647203 0 0) 0x7fdc7813bb90 con 0x7fdc7813c010 2024-09-16T03:23:52.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient(hunting): my global_id is 164323 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7813b6f0 con 0x7fdc7813c010 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2094328176 0 0) 0x7fdc780718e0 con 0x7fdc78135950 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 >> v1:172.21.15.46:6789/0 conn(0x7fdc7813c010 legacy=0x7fdc7806f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 >> v1:172.21.15.45:6789/0 conn(0x7fdc7809fc40 legacy=0x7fdc7806e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 monclient: found mon.c 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7809d2c0 con 0x7fdc78135950 2024-09-16T03:23:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient: _finish_auth 0 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 5 monclient: authenticate success, global_id 184142 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient: _renew_subs 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc78071bc0 con 0x7fdc78135950 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 client.184142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient: _renew_subs 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc781d3b90 con 0x7fdc78135950 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 1 librados: init done 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc68003620 con 0x7fdc78135950 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:52.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fdc680039b0 con 0x7fdc78135950 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc74d99640 4 set_mon_vals no callback set 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 monclient: dump: 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:52.949 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc5c005090 con 0x7fdc78135950 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fdc68003b50 con 0x7fdc78135950 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7fdc6804b410 con 0x7fdc78135950 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 client.184142.objecter ms_dispatch 0x7fdc78000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 3 client.184142.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 3 client.184142.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 20 client.184142.objecter dump_active .. 0 homeless 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc7659c640 10 monclient: get_auth_request con 0x7fdc5803e3b0 auth_method 0 2024-09-16T03:23:52.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.931+0000 7fdc66ffd640 10 client.184142.objecter ms_handle_connect 0x7fdc5803e3b0 2024-09-16T03:23:52.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.935+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fdc68016970 con 0x7fdc78135950 2024-09-16T03:23:52.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.935+0000 7fdc66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:52.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:52.935+0000 7fdc66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:53.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.099+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc5c000ba0 con 0x7fdc5803e3b0 2024-09-16T03:23:53.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.103+0000 7fdc66ffd640 1 -- 172.21.15.45:0/2552751971 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114047 (unknown 2965378022 0 1507276725) 0x7fdc5c000ba0 con 0x7fdc5803e3b0 2024-09-16T03:23:53.106 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:23:53.107 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-16T03:23:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 10 librados: watch_flush enter 2024-09-16T03:23:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 10 librados: watch_flush exit 2024-09-16T03:23:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 20 client.184142.objecter shutdown clearing up homeless session... 2024-09-16T03:23:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 10 client.184142.objecter successfully canceled tick 2024-09-16T03:23:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 >> v1:172.21.15.46:6812/149360280 conn(0x7fdc5803e3b0 legacy=0x7fdc58040870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 10 monclient: shutdown 2024-09-16T03:23:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 >> v1:172.21.15.45:6790/0 conn(0x7fdc78135950 legacy=0x7fdc7806ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 shutdown_connections 2024-09-16T03:23:53.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 >> 172.21.15.45:0/2552751971 conn(0x7fdc780a2bd0 msgr2=0x7fdc7813eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:53.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 shutdown_connections 2024-09-16T03:23:53.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 1 -- 172.21.15.45:0/2552751971 wait complete. 2024-09-16T03:23:53.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.107+0000 7fdc7cf25640 1 librados: shutdown 2024-09-16T03:23:53.123 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1261,"stamp":"2024-09-16T03:23:51.870103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":469008,"kb_used_data":52776,"kb_used_omap":9,"kb_used_meta":416054,"kb_avail":565762032,"statfs":{"total":579820584960,"available":579340320768,"internally_reserved":0,"allocated":54042624,"data_stored":25919678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":426039970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004355"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210772+0000","last_change":"2024-09-16T03:06:02.756614+0000","last_active":"2024-09-16T03:22:27.210772+0000","last_peered":"2024-09-16T03:22:27.210772+0000","last_clean":"2024-09-16T03:22:27.210772+0000","last_became_active":"2024-09-16T03:06:02.756187+0000","last_became_peered":"2024-09-16T03:06:02.756187+0000","last_unstale":"2024-09-16T03:22:27.210772+0000","last_undegraded":"2024-09-16T03:22:27.210772+0000","last_fullsized":"2024-09-16T03:22:27.210772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:43.602727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211989+0000","last_change":"2024-09-16T03:06:02.806920+0000","last_active":"2024-09-16T03:22:27.211989+0000","last_peered":"2024-09-16T03:22:27.211989+0000","last_clean":"2024-09-16T03:22:27.211989+0000","last_became_active":"2024-09-16T03:06:02.798283+0000","last_became_peered":"2024-09-16T03:06:02.798283+0000","last_unstale":"2024-09-16T03:22:27.211989+0000","last_undegraded":"2024-09-16T03:22:27.211989+0000","last_fullsized":"2024-09-16T03:22:27.211989+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:43.018291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.212003+0000","last_change":"2024-09-16T03:06:02.832611+0000","last_active":"2024-09-16T03:22:27.212003+0000","last_peered":"2024-09-16T03:22:27.212003+0000","last_clean":"2024-09-16T03:22:27.212003+0000","last_became_active":"2024-09-16T03:06:02.830437+0000","last_became_peered":"2024-09-16T03:06:02.830437+0000","last_unstale":"2024-09-16T03:22:27.212003+0000","last_undegraded":"2024-09-16T03:22:27.212003+0000","last_fullsized":"2024-09-16T03:22:27.212003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:25.137935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774614+0000","last_change":"2024-09-16T03:06:02.664001+0000","last_active":"2024-09-16T03:22:26.774614+0000","last_peered":"2024-09-16T03:22:26.774614+0000","last_clean":"2024-09-16T03:22:26.774614+0000","last_became_active":"2024-09-16T03:06:02.661502+0000","last_became_peered":"2024-09-16T03:06:02.661502+0000","last_unstale":"2024-09-16T03:22:26.774614+0000","last_undegraded":"2024-09-16T03:22:26.774614+0000","last_fullsized":"2024-09-16T03:22:26.774614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:56.502316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209178+0000","last_change":"2024-09-16T03:06:02.750362+0000","last_active":"2024-09-16T03:22:27.209178+0000","last_peered":"2024-09-16T03:22:27.209178+0000","last_clean":"2024-09-16T03:22:27.209178+0000","last_became_active":"2024-09-16T03:06:02.744921+0000","last_became_peered":"2024-09-16T03:06:02.744921+0000","last_unstale":"2024-09-16T03:22:27.209178+0000","last_undegraded":"2024-09-16T03:22:27.209178+0000","last_fullsized":"2024-09-16T03:22:27.209178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:03.960933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1336,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777944+0000","last_change":"2024-09-16T03:06:02.747722+0000","last_active":"2024-09-16T03:22:26.777944+0000","last_peered":"2024-09-16T03:22:26.777944+0000","last_clean":"2024-09-16T03:22:26.777944+0000","last_became_active":"2024-09-16T03:06:02.746529+0000","last_became_peered":"2024-09-16T03:06:02.746529+0000","last_unstale":"2024-09-16T03:22:26.777944+0000","last_undegraded":"2024-09-16T03:22:26.777944+0000","last_fullsized":"2024-09-16T03:22:26.777944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:54.961886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209716+0000","last_change":"2024-09-16T03:06:02.755623+0000","last_active":"2024-09-16T03:22:27.209716+0000","last_peered":"2024-09-16T03:22:27.209716+0000","last_clean":"2024-09-16T03:22:27.209716+0000","last_became_active":"2024-09-16T03:06:02.755275+0000","last_became_peered":"2024-09-16T03:06:02.755275+0000","last_unstale":"2024-09-16T03:22:27.209716+0000","last_undegraded":"2024-09-16T03:22:27.209716+0000","last_fullsized":"2024-09-16T03:22:27.209716+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:53.461709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122045+0000","last_change":"2024-09-16T03:06:02.820568+0000","last_active":"2024-09-16T03:22:27.122045+0000","last_peered":"2024-09-16T03:22:27.122045+0000","last_clean":"2024-09-16T03:22:27.122045+0000","last_became_active":"2024-09-16T03:06:02.820265+0000","last_became_peered":"2024-09-16T03:06:02.820265+0000","last_unstale":"2024-09-16T03:22:27.122045+0000","last_undegraded":"2024-09-16T03:22:27.122045+0000","last_fullsized":"2024-09-16T03:22:27.122045+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:30.846375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774889+0000","last_change":"2024-09-16T03:06:02.699671+0000","last_active":"2024-09-16T03:22:26.774889+0000","last_peered":"2024-09-16T03:22:26.774889+0000","last_clean":"2024-09-16T03:22:26.774889+0000","last_became_active":"2024-09-16T03:06:02.698531+0000","last_became_peered":"2024-09-16T03:06:02.698531+0000","last_unstale":"2024-09-16T03:22:26.774889+0000","last_undegraded":"2024-09-16T03:22:26.774889+0000","last_fullsized":"2024-09-16T03:22:26.774889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:19:31.192437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777641+0000","last_change":"2024-09-16T03:06:02.829600+0000","last_active":"2024-09-16T03:22:26.777641+0000","last_peered":"2024-09-16T03:22:26.777641+0000","last_clean":"2024-09-16T03:22:26.777641+0000","last_became_active":"2024-09-16T03:06:02.829225+0000","last_became_peered":"2024-09-16T03:06:02.829225+0000","last_unstale":"2024-09-16T03:22:26.777641+0000","last_undegraded":"2024-09-16T03:22:26.777641+0000","last_fullsized":"2024-09-16T03:22:26.777641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:02.011377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211564+0000","last_change":"2024-09-16T03:06:02.755653+0000","last_active":"2024-09-16T03:22:27.211564+0000","last_peered":"2024-09-16T03:22:27.211564+0000","last_clean":"2024-09-16T03:22:27.211564+0000","last_became_active":"2024-09-16T03:06:02.750016+0000","last_became_peered":"2024-09-16T03:06:02.750016+0000","last_unstale":"2024-09-16T03:22:27.211564+0000","last_undegraded":"2024-09-16T03:22:27.211564+0000","last_fullsized":"2024-09-16T03:22:27.211564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:44.526283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777284+0000","last_change":"2024-09-16T03:06:02.858185+0000","last_active":"2024-09-16T03:22:26.777284+0000","last_peered":"2024-09-16T03:22:26.777284+0000","last_clean":"2024-09-16T03:22:26.777284+0000","last_became_active":"2024-09-16T03:06:02.857164+0000","last_became_peered":"2024-09-16T03:06:02.857164+0000","last_unstale":"2024-09-16T03:22:26.777284+0000","last_undegraded":"2024-09-16T03:22:26.777284+0000","last_fullsized":"2024-09-16T03:22:26.777284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:44.538058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1356,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209549+0000","last_change":"2024-09-16T03:04:59.132718+0000","last_active":"2024-09-16T03:22:27.209549+0000","last_peered":"2024-09-16T03:22:27.209549+0000","last_clean":"2024-09-16T03:22:27.209549+0000","last_became_active":"2024-09-16T03:04:57.691813+0000","last_became_peered":"2024-09-16T03:04:57.691813+0000","last_unstale":"2024-09-16T03:22:27.209549+0000","last_undegraded":"2024-09-16T03:22:27.209549+0000","last_fullsized":"2024-09-16T03:22:27.209549+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:55.115695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015712199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210838+0000","last_change":"2024-09-16T03:06:02.721325+0000","last_active":"2024-09-16T03:22:27.210838+0000","last_peered":"2024-09-16T03:22:27.210838+0000","last_clean":"2024-09-16T03:22:27.210838+0000","last_became_active":"2024-09-16T03:06:02.721049+0000","last_became_peered":"2024-09-16T03:06:02.721049+0000","last_unstale":"2024-09-16T03:22:27.210838+0000","last_undegraded":"2024-09-16T03:22:27.210838+0000","last_fullsized":"2024-09-16T03:22:27.210838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:56.615690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1363,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.123007+0000","last_change":"2024-09-16T03:04:59.142318+0000","last_active":"2024-09-16T03:22:27.123007+0000","last_peered":"2024-09-16T03:22:27.123007+0000","last_clean":"2024-09-16T03:22:27.123007+0000","last_became_active":"2024-09-16T03:04:57.117464+0000","last_became_peered":"2024-09-16T03:04:57.117464+0000","last_unstale":"2024-09-16T03:22:27.123007+0000","last_undegraded":"2024-09-16T03:22:27.123007+0000","last_fullsized":"2024-09-16T03:22:27.123007+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:54:40.258398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012748499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209678+0000","last_change":"2024-09-16T03:06:02.769149+0000","last_active":"2024-09-16T03:22:27.209678+0000","last_peered":"2024-09-16T03:22:27.209678+0000","last_clean":"2024-09-16T03:22:27.209678+0000","last_became_active":"2024-09-16T03:06:02.768926+0000","last_became_peered":"2024-09-16T03:06:02.768926+0000","last_unstale":"2024-09-16T03:22:27.209678+0000","last_undegraded":"2024-09-16T03:22:27.209678+0000","last_fullsized":"2024-09-16T03:22:27.209678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:11.356942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1357,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209966+0000","last_change":"2024-09-16T03:04:59.131904+0000","last_active":"2024-09-16T03:22:27.209966+0000","last_peered":"2024-09-16T03:22:27.209966+0000","last_clean":"2024-09-16T03:22:27.209966+0000","last_became_active":"2024-09-16T03:04:57.113180+0000","last_became_peered":"2024-09-16T03:04:57.113180+0000","last_unstale":"2024-09-16T03:22:27.209966+0000","last_undegraded":"2024-09-16T03:22:27.209966+0000","last_fullsized":"2024-09-16T03:22:27.209966+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:30.278408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087237100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1361,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122484+0000","last_change":"2024-09-16T03:06:02.678570+0000","last_active":"2024-09-16T03:22:27.122484+0000","last_peered":"2024-09-16T03:22:27.122484+0000","last_clean":"2024-09-16T03:22:27.122484+0000","last_became_active":"2024-09-16T03:06:02.662210+0000","last_became_peered":"2024-09-16T03:06:02.662210+0000","last_unstale":"2024-09-16T03:22:27.122484+0000","last_undegraded":"2024-09-16T03:22:27.122484+0000","last_fullsized":"2024-09-16T03:22:27.122484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:18.499817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1363,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122893+0000","last_change":"2024-09-16T03:04:59.142273+0000","last_active":"2024-09-16T03:22:27.122893+0000","last_peered":"2024-09-16T03:22:27.122893+0000","last_clean":"2024-09-16T03:22:27.122893+0000","last_became_active":"2024-09-16T03:04:57.117660+0000","last_became_peered":"2024-09-16T03:04:57.117660+0000","last_unstale":"2024-09-16T03:22:27.122893+0000","last_undegraded":"2024-09-16T03:22:27.122893+0000","last_fullsized":"2024-09-16T03:22:27.122893+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:22.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211578+0000","last_change":"2024-09-16T03:06:02.853549+0000","last_active":"2024-09-16T03:22:27.211578+0000","last_peered":"2024-09-16T03:22:27.211578+0000","last_clean":"2024-09-16T03:22:27.211578+0000","last_became_active":"2024-09-16T03:06:02.852993+0000","last_became_peered":"2024-09-16T03:06:02.852993+0000","last_unstale":"2024-09-16T03:22:27.211578+0000","last_undegraded":"2024-09-16T03:22:27.211578+0000","last_fullsized":"2024-09-16T03:22:27.211578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:42.416142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774729+0000","last_change":"2024-09-16T03:04:59.138496+0000","last_active":"2024-09-16T03:22:26.774729+0000","last_peered":"2024-09-16T03:22:26.774729+0000","last_clean":"2024-09-16T03:22:26.774729+0000","last_became_active":"2024-09-16T03:04:57.119695+0000","last_became_peered":"2024-09-16T03:04:57.119695+0000","last_unstale":"2024-09-16T03:22:26.774729+0000","last_undegraded":"2024-09-16T03:22:26.774729+0000","last_fullsized":"2024-09-16T03:22:26.774729+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:24.631318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024209740000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.123006+0000","last_change":"2024-09-16T03:06:02.765932+0000","last_active":"2024-09-16T03:22:27.123006+0000","last_peered":"2024-09-16T03:22:27.123006+0000","last_clean":"2024-09-16T03:22:27.123006+0000","last_became_active":"2024-09-16T03:06:02.760889+0000","last_became_peered":"2024-09-16T03:06:02.760889+0000","last_unstale":"2024-09-16T03:22:27.123006+0000","last_undegraded":"2024-09-16T03:22:27.123006+0000","last_fullsized":"2024-09-16T03:22:27.123006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:26.594180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1352,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777443+0000","last_change":"2024-09-16T03:04:59.133304+0000","last_active":"2024-09-16T03:22:26.777443+0000","last_peered":"2024-09-16T03:22:26.777443+0000","last_clean":"2024-09-16T03:22:26.777443+0000","last_became_active":"2024-09-16T03:04:57.118413+0000","last_became_peered":"2024-09-16T03:04:57.118413+0000","last_unstale":"2024-09-16T03:22:26.777443+0000","last_undegraded":"2024-09-16T03:22:26.777443+0000","last_fullsized":"2024-09-16T03:22:26.777443+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:24.300841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014587720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774981+0000","last_change":"2024-09-16T03:06:02.746071+0000","last_active":"2024-09-16T03:22:26.774981+0000","last_peered":"2024-09-16T03:22:26.774981+0000","last_clean":"2024-09-16T03:22:26.774981+0000","last_became_active":"2024-09-16T03:06:02.743279+0000","last_became_peered":"2024-09-16T03:06:02.743279+0000","last_unstale":"2024-09-16T03:22:26.774981+0000","last_undegraded":"2024-09-16T03:22:26.774981+0000","last_fullsized":"2024-09-16T03:22:26.774981+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:12.479265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1357,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209320+0000","last_change":"2024-09-16T03:04:59.132034+0000","last_active":"2024-09-16T03:22:27.209320+0000","last_peered":"2024-09-16T03:22:27.209320+0000","last_clean":"2024-09-16T03:22:27.209320+0000","last_became_active":"2024-09-16T03:04:57.117048+0000","last_became_peered":"2024-09-16T03:04:57.117048+0000","last_unstale":"2024-09-16T03:22:27.209320+0000","last_undegraded":"2024-09-16T03:22:27.209320+0000","last_fullsized":"2024-09-16T03:22:27.209320+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:57.899748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099232000000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122142+0000","last_change":"2024-09-16T03:06:02.822684+0000","last_active":"2024-09-16T03:22:27.122142+0000","last_peered":"2024-09-16T03:22:27.122142+0000","last_clean":"2024-09-16T03:22:27.122142+0000","last_became_active":"2024-09-16T03:06:02.820554+0000","last_became_peered":"2024-09-16T03:06:02.820554+0000","last_unstale":"2024-09-16T03:22:27.122142+0000","last_undegraded":"2024-09-16T03:22:27.122142+0000","last_fullsized":"2024-09-16T03:22:27.122142+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:24:48.622458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774841+0000","last_change":"2024-09-16T03:04:59.137133+0000","last_active":"2024-09-16T03:22:26.774841+0000","last_peered":"2024-09-16T03:22:26.774841+0000","last_clean":"2024-09-16T03:22:26.774841+0000","last_became_active":"2024-09-16T03:04:57.116849+0000","last_became_peered":"2024-09-16T03:04:57.116849+0000","last_unstale":"2024-09-16T03:22:26.774841+0000","last_undegraded":"2024-09-16T03:22:26.774841+0000","last_fullsized":"2024-09-16T03:22:26.774841+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:51.385856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00103192,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1448,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209486+0000","last_change":"2024-09-16T03:04:51.319467+0000","last_active":"2024-09-16T03:22:27.209486+0000","last_peered":"2024-09-16T03:22:27.209486+0000","last_clean":"2024-09-16T03:22:27.209486+0000","last_became_active":"2024-09-16T03:04:51.317160+0000","last_became_peered":"2024-09-16T03:04:51.317160+0000","last_unstale":"2024-09-16T03:22:27.209486+0000","last_undegraded":"2024-09-16T03:22:27.209486+0000","last_fullsized":"2024-09-16T03:22:27.209486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_clean_scrub_stamp":"2024-09-16T03:04:49.540253+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:15.016218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.775063+0000","last_change":"2024-09-16T03:06:02.762194+0000","last_active":"2024-09-16T03:22:26.775063+0000","last_peered":"2024-09-16T03:22:26.775063+0000","last_clean":"2024-09-16T03:22:26.775063+0000","last_became_active":"2024-09-16T03:06:02.761928+0000","last_became_peered":"2024-09-16T03:06:02.761928+0000","last_unstale":"2024-09-16T03:22:26.775063+0000","last_undegraded":"2024-09-16T03:22:26.775063+0000","last_fullsized":"2024-09-16T03:22:26.775063+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:27:32.863289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774795+0000","last_change":"2024-09-16T03:06:02.664328+0000","last_active":"2024-09-16T03:22:26.774795+0000","last_peered":"2024-09-16T03:22:26.774795+0000","last_clean":"2024-09-16T03:22:26.774795+0000","last_became_active":"2024-09-16T03:06:02.661047+0000","last_became_peered":"2024-09-16T03:06:02.661047+0000","last_unstale":"2024-09-16T03:22:26.774795+0000","last_undegraded":"2024-09-16T03:22:26.774795+0000","last_fullsized":"2024-09-16T03:22:26.774795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:52.055646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1346,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122655+0000","last_change":"2024-09-16T03:06:02.769535+0000","last_active":"2024-09-16T03:22:27.122655+0000","last_peered":"2024-09-16T03:22:27.122655+0000","last_clean":"2024-09-16T03:22:27.122655+0000","last_became_active":"2024-09-16T03:06:02.769042+0000","last_became_peered":"2024-09-16T03:06:02.769042+0000","last_unstale":"2024-09-16T03:22:27.122655+0000","last_undegraded":"2024-09-16T03:22:27.122655+0000","last_fullsized":"2024-09-16T03:22:27.122655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:26.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777928+0000","last_change":"2024-09-16T03:06:02.829591+0000","last_active":"2024-09-16T03:22:26.777928+0000","last_peered":"2024-09-16T03:22:26.777928+0000","last_clean":"2024-09-16T03:22:26.777928+0000","last_became_active":"2024-09-16T03:06:02.829311+0000","last_became_peered":"2024-09-16T03:06:02.829311+0000","last_unstale":"2024-09-16T03:22:26.777928+0000","last_undegraded":"2024-09-16T03:22:26.777928+0000","last_fullsized":"2024-09-16T03:22:26.777928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:04.717263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211030+0000","last_change":"2024-09-16T03:06:02.821360+0000","last_active":"2024-09-16T03:22:27.211030+0000","last_peered":"2024-09-16T03:22:27.211030+0000","last_clean":"2024-09-16T03:22:27.211030+0000","last_became_active":"2024-09-16T03:06:02.820637+0000","last_became_peered":"2024-09-16T03:06:02.820637+0000","last_unstale":"2024-09-16T03:22:27.211030+0000","last_undegraded":"2024-09-16T03:22:27.211030+0000","last_fullsized":"2024-09-16T03:22:27.211030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:56.743641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211911+0000","last_change":"2024-09-16T03:06:02.800273+0000","last_active":"2024-09-16T03:22:27.211911+0000","last_peered":"2024-09-16T03:22:27.211911+0000","last_clean":"2024-09-16T03:22:27.211911+0000","last_became_active":"2024-09-16T03:06:02.798149+0000","last_became_peered":"2024-09-16T03:06:02.798149+0000","last_unstale":"2024-09-16T03:22:27.211911+0000","last_undegraded":"2024-09-16T03:22:27.211911+0000","last_fullsized":"2024-09-16T03:22:27.211911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:44.563168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209139+0000","last_change":"2024-09-16T03:06:02.739124+0000","last_active":"2024-09-16T03:22:27.209139+0000","last_peered":"2024-09-16T03:22:27.209139+0000","last_clean":"2024-09-16T03:22:27.209139+0000","last_became_active":"2024-09-16T03:06:02.736697+0000","last_became_peered":"2024-09-16T03:06:02.736697+0000","last_unstale":"2024-09-16T03:22:27.209139+0000","last_undegraded":"2024-09-16T03:22:27.209139+0000","last_fullsized":"2024-09-16T03:22:27.209139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:56:28.860331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122618+0000","last_change":"2024-09-16T03:06:02.661500+0000","last_active":"2024-09-16T03:22:27.122618+0000","last_peered":"2024-09-16T03:22:27.122618+0000","last_clean":"2024-09-16T03:22:27.122618+0000","last_became_active":"2024-09-16T03:06:02.660659+0000","last_became_peered":"2024-09-16T03:06:02.660659+0000","last_unstale":"2024-09-16T03:22:27.122618+0000","last_undegraded":"2024-09-16T03:22:27.122618+0000","last_fullsized":"2024-09-16T03:22:27.122618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:50.530990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774695+0000","last_change":"2024-09-16T03:06:02.675225+0000","last_active":"2024-09-16T03:22:26.774695+0000","last_peered":"2024-09-16T03:22:26.774695+0000","last_clean":"2024-09-16T03:22:26.774695+0000","last_became_active":"2024-09-16T03:06:02.668156+0000","last_became_peered":"2024-09-16T03:06:02.668156+0000","last_unstale":"2024-09-16T03:22:26.774695+0000","last_undegraded":"2024-09-16T03:22:26.774695+0000","last_fullsized":"2024-09-16T03:22:26.774695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:10:19.961091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209688+0000","last_change":"2024-09-16T03:06:02.755239+0000","last_active":"2024-09-16T03:22:27.209688+0000","last_peered":"2024-09-16T03:22:27.209688+0000","last_clean":"2024-09-16T03:22:27.209688+0000","last_became_active":"2024-09-16T03:06:02.752964+0000","last_became_peered":"2024-09-16T03:06:02.752964+0000","last_unstale":"2024-09-16T03:22:27.209688+0000","last_undegraded":"2024-09-16T03:22:27.209688+0000","last_fullsized":"2024-09-16T03:22:27.209688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:17.029098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777476+0000","last_change":"2024-09-16T03:06:02.749131+0000","last_active":"2024-09-16T03:22:26.777476+0000","last_peered":"2024-09-16T03:22:26.777476+0000","last_clean":"2024-09-16T03:22:26.777476+0000","last_became_active":"2024-09-16T03:06:02.748671+0000","last_became_peered":"2024-09-16T03:06:02.748671+0000","last_unstale":"2024-09-16T03:22:26.777476+0000","last_undegraded":"2024-09-16T03:22:26.777476+0000","last_fullsized":"2024-09-16T03:22:26.777476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.998002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210912+0000","last_change":"2024-09-16T03:06:02.723416+0000","last_active":"2024-09-16T03:22:27.210912+0000","last_peered":"2024-09-16T03:22:27.210912+0000","last_clean":"2024-09-16T03:22:27.210912+0000","last_became_active":"2024-09-16T03:06:02.723112+0000","last_became_peered":"2024-09-16T03:06:02.723112+0000","last_unstale":"2024-09-16T03:22:27.210912+0000","last_undegraded":"2024-09-16T03:22:27.210912+0000","last_fullsized":"2024-09-16T03:22:27.210912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:02.720767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.212250+0000","last_change":"2024-09-16T03:06:02.845346+0000","last_active":"2024-09-16T03:22:27.212250+0000","last_peered":"2024-09-16T03:22:27.212250+0000","last_clean":"2024-09-16T03:22:27.212250+0000","last_became_active":"2024-09-16T03:06:02.843866+0000","last_became_peered":"2024-09-16T03:06:02.843866+0000","last_unstale":"2024-09-16T03:22:27.212250+0000","last_undegraded":"2024-09-16T03:22:27.212250+0000","last_fullsized":"2024-09-16T03:22:27.212250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:45.180460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607782,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84240,"kb_used_data":8624,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287600,"statfs":{"total":96636764160,"available":96550502400,"internally_reserved":0,"allocated":8830976,"data_stored":4143648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607783,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71748,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300092,"statfs":{"total":96636764160,"available":96563294208,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607782,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84752,"kb_used_data":9144,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287088,"statfs":{"total":96636764160,"available":96549978112,"internally_reserved":0,"allocated":9363456,"data_stored":4672562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607782,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71764,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300076,"statfs":{"total":96636764160,"available":96563277824,"internally_reserved":0,"allocated":8835072,"data_stored":4143653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640487,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72276,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94299564,"statfs":{"total":96636764160,"available":96562753536,"internally_reserved":0,"allocated":9359360,"data_stored":4672557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84228,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287612,"statfs":{"total":96636764160,"available":96550514688,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:23:53.125 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:23:53.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:23:53.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient: get_monmap_and_config 2024-09-16T03:23:53.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient: build_initial_monmap 2024-09-16T03:23:53.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient: monmap: 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:53.249636+0000 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:53.249636+0000 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:53.248 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 1 Processor -- start 2024-09-16T03:23:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 1 -- start start 2024-09-16T03:23:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient: init 2024-09-16T03:23:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:23:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient(hunting): picked mon.noname-c con 0x7f58c809c570 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:53.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient(hunting): picked mon.noname-a con 0x7f58c8134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient(hunting): picked mon.noname-b con 0x7f58c81347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c8135150 con 0x7f58c8134320 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c8074470 con 0x7f58c809c570 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c8074750 con 0x7f58c81347d0 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 10 monclient(hunting): authenticate will time out at 2123.899043s 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58ccf42640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f58c8134320 0x7f58c813cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60262/0 (socket says 172.21.15.45:60262) 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58ccf42640 1 -- 172.21.15.45:0/2876130389 learned_addr learned my addr 172.21.15.45:0/2876130389 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cdf44640 1 --1- 172.21.15.45:0/2876130389 >> v1:172.21.15.45:6790/0 conn(0x7f58c81347d0 0x7f58c81402a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58450/0 (socket says 172.21.15.45:58450) 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2622621281 0 0) 0x7f58b8001120 con 0x7f58c8134320 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient(hunting): my global_id is 174246 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58b4003530 con 0x7f58c8134320 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2811351470 0 0) 0x7f58c8074750 con 0x7f58c81347d0 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient(hunting): my global_id is 184145 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58b8001120 con 0x7f58c81347d0 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1216602834 0 0) 0x7f58b4003530 con 0x7f58c8134320 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 >> v1:172.21.15.45:6790/0 conn(0x7f58c81347d0 legacy=0x7f58c81402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 >> v1:172.21.15.46:6789/0 conn(0x7f58c809c570 legacy=0x7f58c809c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient: _finish_hunting 0 2024-09-16T03:23:53.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 monclient: found mon.noname-a 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c8074a30 con 0x7f58c8134320 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient: _finish_auth 0 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient: _check_auth_tickets 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 5 monclient: authenticate success, global_id 174246 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f58b8001870 con 0x7f58c8134320 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58cf9a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.247+0000 7f58bffff640 10 monclient: dump: 2024-09-16T03:23:53.259 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:53.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f58b8001a10 con 0x7f58c8134320 2024-09-16T03:23:53.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:53.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bffff640 1 -- 172.21.15.45:0/2876130389 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f58b80048b0 con 0x7f58c8134320 2024-09-16T03:23:53.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:53.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 4 set_mon_vals no callback set 2024-09-16T03:23:53.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:53.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:53.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bffff640 10 monclient: dump: 2024-09-16T03:23:53.261 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: shutdown 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- 172.21.15.45:0/2876130389 >> v1:172.21.15.45:6789/0 conn(0x7f58c8134320 legacy=0x7f58c813cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- 172.21.15.45:0/2876130389 shutdown_connections 2024-09-16T03:23:53.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- 172.21.15.45:0/2876130389 >> 172.21.15.45:0/2876130389 conn(0x7f58c80a2bd0 msgr2=0x7f58c80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- 172.21.15.45:0/2876130389 shutdown_connections 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- 172.21.15.45:0/2876130389 wait complete. 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: build_initial_monmap 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: monmap: 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:53.263 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 librados: starting msgr at 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 librados: starting objecter 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 Processor -- start 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- start start 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 librados: setting wanted keys 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 librados: calling monclient init 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: init 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient(hunting): picked mon.noname-b con 0x7f58c809c570 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient(hunting): picked mon.noname-a con 0x7f58c8134320 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:53.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient(hunting): picked mon.noname-c con 0x7f58c81347d0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c809d2c0 con 0x7f58c8134320 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c8133bb0 con 0x7f58c809c570 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c8133e90 con 0x7f58c81347d0 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient(hunting): authenticate will time out at 2123.902199s 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58ccf42640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f58c8134320 0x7f58c81d2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60264/0 (socket says 172.21.15.45:60264) 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58ccf42640 1 -- 172.21.15.45:0/1978243505 learned_addr learned my addr 172.21.15.45:0/1978243505 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 client.?.objecter ms_handle_connect 0x7f58c81347d0 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 client.?.objecter ms_handle_connect 0x7f58c8134320 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 999662414 0 0) 0x7f58c8133e90 con 0x7f58c81347d0 2024-09-16T03:23:53.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): my global_id is 184148 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58ac004ec0 con 0x7f58c81347d0 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2011152274 0 0) 0x7f58c8074a30 con 0x7f58c8134320 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): my global_id is 174249 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58c8133e90 con 0x7f58c8134320 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 client.?.objecter ms_handle_connect 0x7f58c809c570 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 240305594 0 0) 0x7f58ac004ec0 con 0x7f58c81347d0 2024-09-16T03:23:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58c8074a30 con 0x7f58c81347d0 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f58c0001940 con 0x7f58c81347d0 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): dump: 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.267 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:53.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 142172521 0 0) 0x7f58c8133e90 con 0x7f58c8134320 2024-09-16T03:23:53.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58ac004ec0 con 0x7f58c8134320 2024-09-16T03:23:53.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f58b8001bc0 con 0x7f58c8134320 2024-09-16T03:23:53.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:53.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): dump: 2024-09-16T03:23:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.270 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847970842 0 0) 0x7f58c8133bb0 con 0x7f58c809c570 2024-09-16T03:23:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient(hunting): my global_id is 164329 2024-09-16T03:23:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58c8133e90 con 0x7f58c809c570 2024-09-16T03:23:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1620620722 0 0) 0x7f58c8074a30 con 0x7f58c81347d0 2024-09-16T03:23:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 >> v1:172.21.15.46:6789/0 conn(0x7f58c809c570 legacy=0x7f58c81cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 >> v1:172.21.15.45:6789/0 conn(0x7f58c8134320 legacy=0x7f58c81d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient: _finish_hunting 0 2024-09-16T03:23:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 monclient: found mon.c 2024-09-16T03:23:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c809b890 con 0x7f58c81347d0 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient: _finish_auth 0 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient: _check_auth_tickets 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 5 monclient: authenticate success, global_id 184148 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: _renew_subs 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f58c80a5920 con 0x7f58c81347d0 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 client.184148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: _renew_subs 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f58c81310d0 con 0x7f58c81347d0 2024-09-16T03:23:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 1 librados: init done 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f58c00047f0 con 0x7f58c81347d0 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f58c0004b40 con 0x7f58c81347d0 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bf7fe640 4 set_mon_vals no callback set 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58bdffb640 10 monclient: dump: 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.274 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.251+0000 7f58cf9a7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.255+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a8005090 con 0x7f58c81347d0 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.255+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f58c0010bb0 con 0x7f58c81347d0 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.255+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f58c004b360 con 0x7f58c81347d0 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.255+0000 7f58bdffb640 10 client.184148.objecter ms_dispatch 0x7f58c8000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.255+0000 7f58bdffb640 3 client.184148.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.255+0000 7f58bdffb640 3 client.184148.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.255+0000 7f58bdffb640 20 client.184148.objecter dump_active .. 0 homeless 2024-09-16T03:23:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.255+0000 7f58cd743640 10 monclient: get_auth_request con 0x7f58ac03e240 auth_method 0 2024-09-16T03:23:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.255+0000 7f58bdffb640 10 client.184148.objecter ms_handle_connect 0x7f58ac03e240 2024-09-16T03:23:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.259+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f58c0016310 con 0x7f58c81347d0 2024-09-16T03:23:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.259+0000 7f58bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.259+0000 7f58bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.419+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f58a8000ba0 con 0x7f58ac03e240 2024-09-16T03:23:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.423+0000 7f58bdffb640 1 -- 172.21.15.45:0/1978243505 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114047 (unknown 2965378022 0 1507276725) 0x7f58a8000ba0 con 0x7f58ac03e240 2024-09-16T03:23:53.425 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:23:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 10 librados: watch_flush enter 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 10 librados: watch_flush exit 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 20 client.184148.objecter shutdown clearing up homeless session... 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 10 client.184148.objecter successfully canceled tick 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 >> v1:172.21.15.46:6812/149360280 conn(0x7f58ac03e240 legacy=0x7f58ac040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 10 monclient: shutdown 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 >> v1:172.21.15.45:6790/0 conn(0x7f58c81347d0 legacy=0x7f58c81d5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 shutdown_connections 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 >> 172.21.15.45:0/1978243505 conn(0x7f58c80a2bd0 msgr2=0x7f58c809f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 shutdown_connections 2024-09-16T03:23:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 1 -- 172.21.15.45:0/1978243505 wait complete. 2024-09-16T03:23:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.427+0000 7f58cf9a7640 1 librados: shutdown 2024-09-16T03:23:53.442 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1261,"stamp":"2024-09-16T03:23:51.870103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":469008,"kb_used_data":52776,"kb_used_omap":9,"kb_used_meta":416054,"kb_avail":565762032,"statfs":{"total":579820584960,"available":579340320768,"internally_reserved":0,"allocated":54042624,"data_stored":25919678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":426039970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004355"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210772+0000","last_change":"2024-09-16T03:06:02.756614+0000","last_active":"2024-09-16T03:22:27.210772+0000","last_peered":"2024-09-16T03:22:27.210772+0000","last_clean":"2024-09-16T03:22:27.210772+0000","last_became_active":"2024-09-16T03:06:02.756187+0000","last_became_peered":"2024-09-16T03:06:02.756187+0000","last_unstale":"2024-09-16T03:22:27.210772+0000","last_undegraded":"2024-09-16T03:22:27.210772+0000","last_fullsized":"2024-09-16T03:22:27.210772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:43.602727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211989+0000","last_change":"2024-09-16T03:06:02.806920+0000","last_active":"2024-09-16T03:22:27.211989+0000","last_peered":"2024-09-16T03:22:27.211989+0000","last_clean":"2024-09-16T03:22:27.211989+0000","last_became_active":"2024-09-16T03:06:02.798283+0000","last_became_peered":"2024-09-16T03:06:02.798283+0000","last_unstale":"2024-09-16T03:22:27.211989+0000","last_undegraded":"2024-09-16T03:22:27.211989+0000","last_fullsized":"2024-09-16T03:22:27.211989+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:43.018291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.212003+0000","last_change":"2024-09-16T03:06:02.832611+0000","last_active":"2024-09-16T03:22:27.212003+0000","last_peered":"2024-09-16T03:22:27.212003+0000","last_clean":"2024-09-16T03:22:27.212003+0000","last_became_active":"2024-09-16T03:06:02.830437+0000","last_became_peered":"2024-09-16T03:06:02.830437+0000","last_unstale":"2024-09-16T03:22:27.212003+0000","last_undegraded":"2024-09-16T03:22:27.212003+0000","last_fullsized":"2024-09-16T03:22:27.212003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:25.137935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774614+0000","last_change":"2024-09-16T03:06:02.664001+0000","last_active":"2024-09-16T03:22:26.774614+0000","last_peered":"2024-09-16T03:22:26.774614+0000","last_clean":"2024-09-16T03:22:26.774614+0000","last_became_active":"2024-09-16T03:06:02.661502+0000","last_became_peered":"2024-09-16T03:06:02.661502+0000","last_unstale":"2024-09-16T03:22:26.774614+0000","last_undegraded":"2024-09-16T03:22:26.774614+0000","last_fullsized":"2024-09-16T03:22:26.774614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:56.502316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209178+0000","last_change":"2024-09-16T03:06:02.750362+0000","last_active":"2024-09-16T03:22:27.209178+0000","last_peered":"2024-09-16T03:22:27.209178+0000","last_clean":"2024-09-16T03:22:27.209178+0000","last_became_active":"2024-09-16T03:06:02.744921+0000","last_became_peered":"2024-09-16T03:06:02.744921+0000","last_unstale":"2024-09-16T03:22:27.209178+0000","last_undegraded":"2024-09-16T03:22:27.209178+0000","last_fullsized":"2024-09-16T03:22:27.209178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:03.960933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1336,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777944+0000","last_change":"2024-09-16T03:06:02.747722+0000","last_active":"2024-09-16T03:22:26.777944+0000","last_peered":"2024-09-16T03:22:26.777944+0000","last_clean":"2024-09-16T03:22:26.777944+0000","last_became_active":"2024-09-16T03:06:02.746529+0000","last_became_peered":"2024-09-16T03:06:02.746529+0000","last_unstale":"2024-09-16T03:22:26.777944+0000","last_undegraded":"2024-09-16T03:22:26.777944+0000","last_fullsized":"2024-09-16T03:22:26.777944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:54.961886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209716+0000","last_change":"2024-09-16T03:06:02.755623+0000","last_active":"2024-09-16T03:22:27.209716+0000","last_peered":"2024-09-16T03:22:27.209716+0000","last_clean":"2024-09-16T03:22:27.209716+0000","last_became_active":"2024-09-16T03:06:02.755275+0000","last_became_peered":"2024-09-16T03:06:02.755275+0000","last_unstale":"2024-09-16T03:22:27.209716+0000","last_undegraded":"2024-09-16T03:22:27.209716+0000","last_fullsized":"2024-09-16T03:22:27.209716+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:53.461709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122045+0000","last_change":"2024-09-16T03:06:02.820568+0000","last_active":"2024-09-16T03:22:27.122045+0000","last_peered":"2024-09-16T03:22:27.122045+0000","last_clean":"2024-09-16T03:22:27.122045+0000","last_became_active":"2024-09-16T03:06:02.820265+0000","last_became_peered":"2024-09-16T03:06:02.820265+0000","last_unstale":"2024-09-16T03:22:27.122045+0000","last_undegraded":"2024-09-16T03:22:27.122045+0000","last_fullsized":"2024-09-16T03:22:27.122045+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:30.846375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774889+0000","last_change":"2024-09-16T03:06:02.699671+0000","last_active":"2024-09-16T03:22:26.774889+0000","last_peered":"2024-09-16T03:22:26.774889+0000","last_clean":"2024-09-16T03:22:26.774889+0000","last_became_active":"2024-09-16T03:06:02.698531+0000","last_became_peered":"2024-09-16T03:06:02.698531+0000","last_unstale":"2024-09-16T03:22:26.774889+0000","last_undegraded":"2024-09-16T03:22:26.774889+0000","last_fullsized":"2024-09-16T03:22:26.774889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:19:31.192437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777641+0000","last_change":"2024-09-16T03:06:02.829600+0000","last_active":"2024-09-16T03:22:26.777641+0000","last_peered":"2024-09-16T03:22:26.777641+0000","last_clean":"2024-09-16T03:22:26.777641+0000","last_became_active":"2024-09-16T03:06:02.829225+0000","last_became_peered":"2024-09-16T03:06:02.829225+0000","last_unstale":"2024-09-16T03:22:26.777641+0000","last_undegraded":"2024-09-16T03:22:26.777641+0000","last_fullsized":"2024-09-16T03:22:26.777641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:02.011377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211564+0000","last_change":"2024-09-16T03:06:02.755653+0000","last_active":"2024-09-16T03:22:27.211564+0000","last_peered":"2024-09-16T03:22:27.211564+0000","last_clean":"2024-09-16T03:22:27.211564+0000","last_became_active":"2024-09-16T03:06:02.750016+0000","last_became_peered":"2024-09-16T03:06:02.750016+0000","last_unstale":"2024-09-16T03:22:27.211564+0000","last_undegraded":"2024-09-16T03:22:27.211564+0000","last_fullsized":"2024-09-16T03:22:27.211564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:44.526283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777284+0000","last_change":"2024-09-16T03:06:02.858185+0000","last_active":"2024-09-16T03:22:26.777284+0000","last_peered":"2024-09-16T03:22:26.777284+0000","last_clean":"2024-09-16T03:22:26.777284+0000","last_became_active":"2024-09-16T03:06:02.857164+0000","last_became_peered":"2024-09-16T03:06:02.857164+0000","last_unstale":"2024-09-16T03:22:26.777284+0000","last_undegraded":"2024-09-16T03:22:26.777284+0000","last_fullsized":"2024-09-16T03:22:26.777284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:44.538058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1356,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209549+0000","last_change":"2024-09-16T03:04:59.132718+0000","last_active":"2024-09-16T03:22:27.209549+0000","last_peered":"2024-09-16T03:22:27.209549+0000","last_clean":"2024-09-16T03:22:27.209549+0000","last_became_active":"2024-09-16T03:04:57.691813+0000","last_became_peered":"2024-09-16T03:04:57.691813+0000","last_unstale":"2024-09-16T03:22:27.209549+0000","last_undegraded":"2024-09-16T03:22:27.209549+0000","last_fullsized":"2024-09-16T03:22:27.209549+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:55.115695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015712199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210838+0000","last_change":"2024-09-16T03:06:02.721325+0000","last_active":"2024-09-16T03:22:27.210838+0000","last_peered":"2024-09-16T03:22:27.210838+0000","last_clean":"2024-09-16T03:22:27.210838+0000","last_became_active":"2024-09-16T03:06:02.721049+0000","last_became_peered":"2024-09-16T03:06:02.721049+0000","last_unstale":"2024-09-16T03:22:27.210838+0000","last_undegraded":"2024-09-16T03:22:27.210838+0000","last_fullsized":"2024-09-16T03:22:27.210838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:56.615690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1363,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.123007+0000","last_change":"2024-09-16T03:04:59.142318+0000","last_active":"2024-09-16T03:22:27.123007+0000","last_peered":"2024-09-16T03:22:27.123007+0000","last_clean":"2024-09-16T03:22:27.123007+0000","last_became_active":"2024-09-16T03:04:57.117464+0000","last_became_peered":"2024-09-16T03:04:57.117464+0000","last_unstale":"2024-09-16T03:22:27.123007+0000","last_undegraded":"2024-09-16T03:22:27.123007+0000","last_fullsized":"2024-09-16T03:22:27.123007+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:54:40.258398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012748499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209678+0000","last_change":"2024-09-16T03:06:02.769149+0000","last_active":"2024-09-16T03:22:27.209678+0000","last_peered":"2024-09-16T03:22:27.209678+0000","last_clean":"2024-09-16T03:22:27.209678+0000","last_became_active":"2024-09-16T03:06:02.768926+0000","last_became_peered":"2024-09-16T03:06:02.768926+0000","last_unstale":"2024-09-16T03:22:27.209678+0000","last_undegraded":"2024-09-16T03:22:27.209678+0000","last_fullsized":"2024-09-16T03:22:27.209678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:11.356942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1357,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209966+0000","last_change":"2024-09-16T03:04:59.131904+0000","last_active":"2024-09-16T03:22:27.209966+0000","last_peered":"2024-09-16T03:22:27.209966+0000","last_clean":"2024-09-16T03:22:27.209966+0000","last_became_active":"2024-09-16T03:04:57.113180+0000","last_became_peered":"2024-09-16T03:04:57.113180+0000","last_unstale":"2024-09-16T03:22:27.209966+0000","last_undegraded":"2024-09-16T03:22:27.209966+0000","last_fullsized":"2024-09-16T03:22:27.209966+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:30.278408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087237100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1361,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122484+0000","last_change":"2024-09-16T03:06:02.678570+0000","last_active":"2024-09-16T03:22:27.122484+0000","last_peered":"2024-09-16T03:22:27.122484+0000","last_clean":"2024-09-16T03:22:27.122484+0000","last_became_active":"2024-09-16T03:06:02.662210+0000","last_became_peered":"2024-09-16T03:06:02.662210+0000","last_unstale":"2024-09-16T03:22:27.122484+0000","last_undegraded":"2024-09-16T03:22:27.122484+0000","last_fullsized":"2024-09-16T03:22:27.122484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:18.499817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1363,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122893+0000","last_change":"2024-09-16T03:04:59.142273+0000","last_active":"2024-09-16T03:22:27.122893+0000","last_peered":"2024-09-16T03:22:27.122893+0000","last_clean":"2024-09-16T03:22:27.122893+0000","last_became_active":"2024-09-16T03:04:57.117660+0000","last_became_peered":"2024-09-16T03:04:57.117660+0000","last_unstale":"2024-09-16T03:22:27.122893+0000","last_undegraded":"2024-09-16T03:22:27.122893+0000","last_fullsized":"2024-09-16T03:22:27.122893+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:22.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211578+0000","last_change":"2024-09-16T03:06:02.853549+0000","last_active":"2024-09-16T03:22:27.211578+0000","last_peered":"2024-09-16T03:22:27.211578+0000","last_clean":"2024-09-16T03:22:27.211578+0000","last_became_active":"2024-09-16T03:06:02.852993+0000","last_became_peered":"2024-09-16T03:06:02.852993+0000","last_unstale":"2024-09-16T03:22:27.211578+0000","last_undegraded":"2024-09-16T03:22:27.211578+0000","last_fullsized":"2024-09-16T03:22:27.211578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:42.416142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774729+0000","last_change":"2024-09-16T03:04:59.138496+0000","last_active":"2024-09-16T03:22:26.774729+0000","last_peered":"2024-09-16T03:22:26.774729+0000","last_clean":"2024-09-16T03:22:26.774729+0000","last_became_active":"2024-09-16T03:04:57.119695+0000","last_became_peered":"2024-09-16T03:04:57.119695+0000","last_unstale":"2024-09-16T03:22:26.774729+0000","last_undegraded":"2024-09-16T03:22:26.774729+0000","last_fullsized":"2024-09-16T03:22:26.774729+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:24.631318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024209740000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.123006+0000","last_change":"2024-09-16T03:06:02.765932+0000","last_active":"2024-09-16T03:22:27.123006+0000","last_peered":"2024-09-16T03:22:27.123006+0000","last_clean":"2024-09-16T03:22:27.123006+0000","last_became_active":"2024-09-16T03:06:02.760889+0000","last_became_peered":"2024-09-16T03:06:02.760889+0000","last_unstale":"2024-09-16T03:22:27.123006+0000","last_undegraded":"2024-09-16T03:22:27.123006+0000","last_fullsized":"2024-09-16T03:22:27.123006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:26.594180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1352,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777443+0000","last_change":"2024-09-16T03:04:59.133304+0000","last_active":"2024-09-16T03:22:26.777443+0000","last_peered":"2024-09-16T03:22:26.777443+0000","last_clean":"2024-09-16T03:22:26.777443+0000","last_became_active":"2024-09-16T03:04:57.118413+0000","last_became_peered":"2024-09-16T03:04:57.118413+0000","last_unstale":"2024-09-16T03:22:26.777443+0000","last_undegraded":"2024-09-16T03:22:26.777443+0000","last_fullsized":"2024-09-16T03:22:26.777443+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:24.300841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014587720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774981+0000","last_change":"2024-09-16T03:06:02.746071+0000","last_active":"2024-09-16T03:22:26.774981+0000","last_peered":"2024-09-16T03:22:26.774981+0000","last_clean":"2024-09-16T03:22:26.774981+0000","last_became_active":"2024-09-16T03:06:02.743279+0000","last_became_peered":"2024-09-16T03:06:02.743279+0000","last_unstale":"2024-09-16T03:22:26.774981+0000","last_undegraded":"2024-09-16T03:22:26.774981+0000","last_fullsized":"2024-09-16T03:22:26.774981+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:12.479265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1357,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209320+0000","last_change":"2024-09-16T03:04:59.132034+0000","last_active":"2024-09-16T03:22:27.209320+0000","last_peered":"2024-09-16T03:22:27.209320+0000","last_clean":"2024-09-16T03:22:27.209320+0000","last_became_active":"2024-09-16T03:04:57.117048+0000","last_became_peered":"2024-09-16T03:04:57.117048+0000","last_unstale":"2024-09-16T03:22:27.209320+0000","last_undegraded":"2024-09-16T03:22:27.209320+0000","last_fullsized":"2024-09-16T03:22:27.209320+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:57.899748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099232000000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122142+0000","last_change":"2024-09-16T03:06:02.822684+0000","last_active":"2024-09-16T03:22:27.122142+0000","last_peered":"2024-09-16T03:22:27.122142+0000","last_clean":"2024-09-16T03:22:27.122142+0000","last_became_active":"2024-09-16T03:06:02.820554+0000","last_became_peered":"2024-09-16T03:06:02.820554+0000","last_unstale":"2024-09-16T03:22:27.122142+0000","last_undegraded":"2024-09-16T03:22:27.122142+0000","last_fullsized":"2024-09-16T03:22:27.122142+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:24:48.622458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774841+0000","last_change":"2024-09-16T03:04:59.137133+0000","last_active":"2024-09-16T03:22:26.774841+0000","last_peered":"2024-09-16T03:22:26.774841+0000","last_clean":"2024-09-16T03:22:26.774841+0000","last_became_active":"2024-09-16T03:04:57.116849+0000","last_became_peered":"2024-09-16T03:04:57.116849+0000","last_unstale":"2024-09-16T03:22:26.774841+0000","last_undegraded":"2024-09-16T03:22:26.774841+0000","last_fullsized":"2024-09-16T03:22:26.774841+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:51.385856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00103192,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1448,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209486+0000","last_change":"2024-09-16T03:04:51.319467+0000","last_active":"2024-09-16T03:22:27.209486+0000","last_peered":"2024-09-16T03:22:27.209486+0000","last_clean":"2024-09-16T03:22:27.209486+0000","last_became_active":"2024-09-16T03:04:51.317160+0000","last_became_peered":"2024-09-16T03:04:51.317160+0000","last_unstale":"2024-09-16T03:22:27.209486+0000","last_undegraded":"2024-09-16T03:22:27.209486+0000","last_fullsized":"2024-09-16T03:22:27.209486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_clean_scrub_stamp":"2024-09-16T03:04:49.540253+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:15.016218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.775063+0000","last_change":"2024-09-16T03:06:02.762194+0000","last_active":"2024-09-16T03:22:26.775063+0000","last_peered":"2024-09-16T03:22:26.775063+0000","last_clean":"2024-09-16T03:22:26.775063+0000","last_became_active":"2024-09-16T03:06:02.761928+0000","last_became_peered":"2024-09-16T03:06:02.761928+0000","last_unstale":"2024-09-16T03:22:26.775063+0000","last_undegraded":"2024-09-16T03:22:26.775063+0000","last_fullsized":"2024-09-16T03:22:26.775063+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:27:32.863289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774795+0000","last_change":"2024-09-16T03:06:02.664328+0000","last_active":"2024-09-16T03:22:26.774795+0000","last_peered":"2024-09-16T03:22:26.774795+0000","last_clean":"2024-09-16T03:22:26.774795+0000","last_became_active":"2024-09-16T03:06:02.661047+0000","last_became_peered":"2024-09-16T03:06:02.661047+0000","last_unstale":"2024-09-16T03:22:26.774795+0000","last_undegraded":"2024-09-16T03:22:26.774795+0000","last_fullsized":"2024-09-16T03:22:26.774795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:52.055646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1346,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122655+0000","last_change":"2024-09-16T03:06:02.769535+0000","last_active":"2024-09-16T03:22:27.122655+0000","last_peered":"2024-09-16T03:22:27.122655+0000","last_clean":"2024-09-16T03:22:27.122655+0000","last_became_active":"2024-09-16T03:06:02.769042+0000","last_became_peered":"2024-09-16T03:06:02.769042+0000","last_unstale":"2024-09-16T03:22:27.122655+0000","last_undegraded":"2024-09-16T03:22:27.122655+0000","last_fullsized":"2024-09-16T03:22:27.122655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:26.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777928+0000","last_change":"2024-09-16T03:06:02.829591+0000","last_active":"2024-09-16T03:22:26.777928+0000","last_peered":"2024-09-16T03:22:26.777928+0000","last_clean":"2024-09-16T03:22:26.777928+0000","last_became_active":"2024-09-16T03:06:02.829311+0000","last_became_peered":"2024-09-16T03:06:02.829311+0000","last_unstale":"2024-09-16T03:22:26.777928+0000","last_undegraded":"2024-09-16T03:22:26.777928+0000","last_fullsized":"2024-09-16T03:22:26.777928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:04.717263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211030+0000","last_change":"2024-09-16T03:06:02.821360+0000","last_active":"2024-09-16T03:22:27.211030+0000","last_peered":"2024-09-16T03:22:27.211030+0000","last_clean":"2024-09-16T03:22:27.211030+0000","last_became_active":"2024-09-16T03:06:02.820637+0000","last_became_peered":"2024-09-16T03:06:02.820637+0000","last_unstale":"2024-09-16T03:22:27.211030+0000","last_undegraded":"2024-09-16T03:22:27.211030+0000","last_fullsized":"2024-09-16T03:22:27.211030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:56.743641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211911+0000","last_change":"2024-09-16T03:06:02.800273+0000","last_active":"2024-09-16T03:22:27.211911+0000","last_peered":"2024-09-16T03:22:27.211911+0000","last_clean":"2024-09-16T03:22:27.211911+0000","last_became_active":"2024-09-16T03:06:02.798149+0000","last_became_peered":"2024-09-16T03:06:02.798149+0000","last_unstale":"2024-09-16T03:22:27.211911+0000","last_undegraded":"2024-09-16T03:22:27.211911+0000","last_fullsized":"2024-09-16T03:22:27.211911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:44.563168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209139+0000","last_change":"2024-09-16T03:06:02.739124+0000","last_active":"2024-09-16T03:22:27.209139+0000","last_peered":"2024-09-16T03:22:27.209139+0000","last_clean":"2024-09-16T03:22:27.209139+0000","last_became_active":"2024-09-16T03:06:02.736697+0000","last_became_peered":"2024-09-16T03:06:02.736697+0000","last_unstale":"2024-09-16T03:22:27.209139+0000","last_undegraded":"2024-09-16T03:22:27.209139+0000","last_fullsized":"2024-09-16T03:22:27.209139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:56:28.860331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122618+0000","last_change":"2024-09-16T03:06:02.661500+0000","last_active":"2024-09-16T03:22:27.122618+0000","last_peered":"2024-09-16T03:22:27.122618+0000","last_clean":"2024-09-16T03:22:27.122618+0000","last_became_active":"2024-09-16T03:06:02.660659+0000","last_became_peered":"2024-09-16T03:06:02.660659+0000","last_unstale":"2024-09-16T03:22:27.122618+0000","last_undegraded":"2024-09-16T03:22:27.122618+0000","last_fullsized":"2024-09-16T03:22:27.122618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:50.530990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774695+0000","last_change":"2024-09-16T03:06:02.675225+0000","last_active":"2024-09-16T03:22:26.774695+0000","last_peered":"2024-09-16T03:22:26.774695+0000","last_clean":"2024-09-16T03:22:26.774695+0000","last_became_active":"2024-09-16T03:06:02.668156+0000","last_became_peered":"2024-09-16T03:06:02.668156+0000","last_unstale":"2024-09-16T03:22:26.774695+0000","last_undegraded":"2024-09-16T03:22:26.774695+0000","last_fullsized":"2024-09-16T03:22:26.774695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:10:19.961091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209688+0000","last_change":"2024-09-16T03:06:02.755239+0000","last_active":"2024-09-16T03:22:27.209688+0000","last_peered":"2024-09-16T03:22:27.209688+0000","last_clean":"2024-09-16T03:22:27.209688+0000","last_became_active":"2024-09-16T03:06:02.752964+0000","last_became_peered":"2024-09-16T03:06:02.752964+0000","last_unstale":"2024-09-16T03:22:27.209688+0000","last_undegraded":"2024-09-16T03:22:27.209688+0000","last_fullsized":"2024-09-16T03:22:27.209688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:17.029098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777476+0000","last_change":"2024-09-16T03:06:02.749131+0000","last_active":"2024-09-16T03:22:26.777476+0000","last_peered":"2024-09-16T03:22:26.777476+0000","last_clean":"2024-09-16T03:22:26.777476+0000","last_became_active":"2024-09-16T03:06:02.748671+0000","last_became_peered":"2024-09-16T03:06:02.748671+0000","last_unstale":"2024-09-16T03:22:26.777476+0000","last_undegraded":"2024-09-16T03:22:26.777476+0000","last_fullsized":"2024-09-16T03:22:26.777476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.998002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210912+0000","last_change":"2024-09-16T03:06:02.723416+0000","last_active":"2024-09-16T03:22:27.210912+0000","last_peered":"2024-09-16T03:22:27.210912+0000","last_clean":"2024-09-16T03:22:27.210912+0000","last_became_active":"2024-09-16T03:06:02.723112+0000","last_became_peered":"2024-09-16T03:06:02.723112+0000","last_unstale":"2024-09-16T03:22:27.210912+0000","last_undegraded":"2024-09-16T03:22:27.210912+0000","last_fullsized":"2024-09-16T03:22:27.210912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:02.720767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.212250+0000","last_change":"2024-09-16T03:06:02.845346+0000","last_active":"2024-09-16T03:22:27.212250+0000","last_peered":"2024-09-16T03:22:27.212250+0000","last_clean":"2024-09-16T03:22:27.212250+0000","last_became_active":"2024-09-16T03:06:02.843866+0000","last_became_peered":"2024-09-16T03:06:02.843866+0000","last_unstale":"2024-09-16T03:22:27.212250+0000","last_undegraded":"2024-09-16T03:22:27.212250+0000","last_fullsized":"2024-09-16T03:22:27.212250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:45.180460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607782,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84240,"kb_used_data":8624,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287600,"statfs":{"total":96636764160,"available":96550502400,"internally_reserved":0,"allocated":8830976,"data_stored":4143648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607783,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71748,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300092,"statfs":{"total":96636764160,"available":96563294208,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607782,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84752,"kb_used_data":9144,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287088,"statfs":{"total":96636764160,"available":96549978112,"internally_reserved":0,"allocated":9363456,"data_stored":4672562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607782,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71764,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300076,"statfs":{"total":96636764160,"available":96563277824,"internally_reserved":0,"allocated":8835072,"data_stored":4143653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640487,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72276,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94299564,"statfs":{"total":96636764160,"available":96562753536,"internally_reserved":0,"allocated":9359360,"data_stored":4672557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84228,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287612,"statfs":{"total":96636764160,"available":96550514688,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:23:53.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: get_monmap_and_config 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: build_initial_monmap 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: monmap: 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:53.556314+0000 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:53.556314+0000 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:53.555 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:53.555 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 Processor -- start 2024-09-16T03:23:53.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- start start 2024-09-16T03:23:53.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: init 2024-09-16T03:23:53.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:53.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:53.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient(hunting): picked mon.noname-b con 0x7fed88137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:53.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient(hunting): picked mon.noname-c con 0x7fed88138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:53.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient(hunting): picked mon.noname-a con 0x7fed8813e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed880a4f90 con 0x7fed8813e2a0 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed880a4cb0 con 0x7fed88138100 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed8809dc70 con 0x7fed88137d20 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient(hunting): authenticate will time out at 2124.205316s 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed877fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fed88137d20 0x7fed880a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58462/0 (socket says 172.21.15.45:58462) 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed877fe640 1 -- 172.21.15.45:0/3127528118 learned_addr learned my addr 172.21.15.45:0/3127528118 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3263914053 0 0) 0x7fed880a4f90 con 0x7fed8813e2a0 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient(hunting): my global_id is 174252 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed70003530 con 0x7fed8813e2a0 2024-09-16T03:23:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452387373 0 0) 0x7fed8809dc70 con 0x7fed88137d20 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient(hunting): my global_id is 184151 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed880a4f90 con 0x7fed88137d20 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2563648419 0 0) 0x7fed880a4f90 con 0x7fed88137d20 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 >> v1:172.21.15.46:6789/0 conn(0x7fed88138100 legacy=0x7fed8813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 >> v1:172.21.15.45:6789/0 conn(0x7fed8813e2a0 legacy=0x7fed88140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: _finish_hunting 0 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 monclient: found mon.noname-b 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8809cc50 con 0x7fed88137d20 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: _finish_auth 0 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: _check_auth_tickets 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 5 monclient: authenticate success, global_id 184151 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fed80001600 con 0x7fed88137d20 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: dump: 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed800017a0 con 0x7fed88137d20 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 1 -- 172.21.15.45:0/3127528118 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fed80001bb0 con 0x7fed88137d20 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed867fc640 10 monclient: dump: 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 4 set_mon_vals no callback set 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: shutdown 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- 172.21.15.45:0/3127528118 >> v1:172.21.15.45:6790/0 conn(0x7fed88137d20 legacy=0x7fed880a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- 172.21.15.45:0/3127528118 shutdown_connections 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- 172.21.15.45:0/3127528118 >> 172.21.15.45:0/3127528118 conn(0x7fed880aa110 msgr2=0x7fed880aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- 172.21.15.45:0/3127528118 shutdown_connections 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- 172.21.15.45:0/3127528118 wait complete. 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: build_initial_monmap 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: monmap: 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 librados: starting msgr at 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 librados: starting objecter 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 Processor -- start 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 -- start start 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 librados: setting wanted keys 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 1 librados: calling monclient init 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.555+0000 7fed8dfbd640 10 monclient: init 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient(hunting): picked mon.noname-c con 0x7fed88137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient(hunting): picked mon.noname-a con 0x7fed88138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:53.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient(hunting): picked mon.noname-b con 0x7fed8813e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed881419d0 con 0x7fed88138100 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed880a1300 con 0x7fed8813e2a0 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed8813a7b0 con 0x7fed88137d20 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient(hunting): authenticate will time out at 2124.208285s 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed877fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fed88137d20 0x7fed880a2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58468/0 (socket says 172.21.15.45:58468) 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed877fe640 1 -- 172.21.15.45:0/4217136811 learned_addr learned my addr 172.21.15.45:0/4217136811 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed86ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fed88138100 0x7fed880a3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60280/0 (socket says 172.21.15.45:60280) 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 client.?.objecter ms_handle_connect 0x7fed88137d20 2024-09-16T03:23:53.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 client.?.objecter ms_handle_connect 0x7fed88138100 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3786469713 0 0) 0x7fed8813a7b0 con 0x7fed88137d20 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): my global_id is 184154 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed74004ec0 con 0x7fed88137d20 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3654767146 0 0) 0x7fed780015c0 con 0x7fed88138100 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): my global_id is 174255 2024-09-16T03:23:53.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed8813a7b0 con 0x7fed88138100 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 client.?.objecter ms_handle_connect 0x7fed8813e2a0 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4084055995 0 0) 0x7fed74004ec0 con 0x7fed88137d20 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed780015c0 con 0x7fed88137d20 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fed800047f0 con 0x7fed88137d20 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): dump: 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1224495957 0 0) 0x7fed8813a7b0 con 0x7fed88138100 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed74004ec0 con 0x7fed88138100 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fed78001e50 con 0x7fed88138100 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): dump: 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.573 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170827094 0 0) 0x7fed7c000ff0 con 0x7fed8813e2a0 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient(hunting): my global_id is 164335 2024-09-16T03:23:53.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed8813a7b0 con 0x7fed8813e2a0 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3899748749 0 0) 0x7fed780015c0 con 0x7fed88137d20 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 >> v1:172.21.15.46:6789/0 conn(0x7fed8813e2a0 legacy=0x7fed880a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 >> v1:172.21.15.45:6789/0 conn(0x7fed88138100 legacy=0x7fed880a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 monclient: found mon.c 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8813a4d0 con 0x7fed88137d20 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient: _finish_auth 0 2024-09-16T03:23:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 5 monclient: authenticate success, global_id 184154 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient: _renew_subs 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fed881d5d80 con 0x7fed88137d20 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 client.184154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient: _renew_subs 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fed881d6120 con 0x7fed88137d20 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 1 librados: init done 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed800016c0 con 0x7fed88137d20 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fed80004de0 con 0x7fed88137d20 2024-09-16T03:23:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed85ffb640 4 set_mon_vals no callback set 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 monclient: dump: 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fed80010e50 con 0x7fed88137d20 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed68005090 con 0x7fed88137d20 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7fed8004b590 con 0x7fed88137d20 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 client.184154.objecter ms_dispatch 0x7fed88000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 3 client.184154.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 3 client.184154.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 20 client.184154.objecter dump_active .. 0 homeless 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed86ffd640 10 monclient: get_auth_request con 0x7fed7403e260 auth_method 0 2024-09-16T03:23:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.559+0000 7fed6ffff640 10 client.184154.objecter ms_handle_connect 0x7fed7403e260 2024-09-16T03:23:53.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.563+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fed80016b90 con 0x7fed88137d20 2024-09-16T03:23:53.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.563+0000 7fed6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:53.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.563+0000 7fed6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:53.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.735+0000 7fed8dfbd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:23:53.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.735+0000 7fed8dfbd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:23:53.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.735+0000 7fed8dfbd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:53.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.735+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed68005380 con 0x7fed88137d20 2024-09-16T03:23:53.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.735+0000 7fed6ffff640 1 -- 172.21.15.45:0/4217136811 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v703) ==== 74+0+14000 (unknown 581356798 0 2744975896) 0x7fed80016490 con 0x7fed88137d20 2024-09-16T03:23:53.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.735+0000 7fed6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:23:53.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.735+0000 7fed6ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:23:53.738 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:23:53.738 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":703,"fsid":"16b8bfd8-03c6-4432-90b7-8e9932b37929","created":"2024-09-16T03:04:38.421437+0000","modified":"2024-09-16T03:22:26.770669+0000","last_up_change":"2024-09-16T03:04:48.525305+0000","last_in_change":"2024-09-16T03:04:40.127320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":101,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:04:49.504411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:04:55.369938+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-16T03:06:01.555251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60a0a607-fe6c-4bd5-b369-e0dd6f6bd551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6800","nonce":3131549395}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6801","nonce":3131549395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6803","nonce":3131549395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6802","nonce":3131549395}]},"public_addr":"172.21.15.45:6800/3131549395","cluster_addr":"172.21.15.45:6801/3131549395","heartbeat_back_addr":"172.21.15.45:6803/3131549395","heartbeat_front_addr":"172.21.15.45:6802/3131549395","state":["exists","up"]},{"osd":1,"uuid":"10adca9c-3542-4901-a439-a1c9e9fcb8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6808","nonce":3228817795}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6809","nonce":3228817795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6811","nonce":3228817795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6810","nonce":3228817795}]},"public_addr":"172.21.15.45:6808/3228817795","cluster_addr":"172.21.15.45:6809/3228817795","heartbeat_back_addr":"172.21.15.45:6811/3228817795","heartbeat_front_addr":"172.21.15.45:6810/3228817795","state":["exists","up"]},{"osd":2,"uuid":"88fe1895-098a-4f13-95b2-4ead1739e71c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6804","nonce":207876738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6805","nonce":207876738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6807","nonce":207876738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.45:6806","nonce":207876738}]},"public_addr":"172.21.15.45:6804/207876738","cluster_addr":"172.21.15.45:6805/207876738","heartbeat_back_addr":"172.21.15.45:6807/207876738","heartbeat_front_addr":"172.21.15.45:6806/207876738","state":["exists","up"]},{"osd":3,"uuid":"b04f94bc-5631-478e-904c-1dfc9a673a7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6808","nonce":3088806475}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6809","nonce":3088806475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6811","nonce":3088806475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6810","nonce":3088806475}]},"public_addr":"172.21.15.46:6808/3088806475","cluster_addr":"172.21.15.46:6809/3088806475","heartbeat_back_addr":"172.21.15.46:6811/3088806475","heartbeat_front_addr":"172.21.15.46:6810/3088806475","state":["exists","up"]},{"osd":4,"uuid":"c2dbca96-afb9-48ed-8a71-8a2bea37d152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6800","nonce":264199318}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":264199318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":264199318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":264199318}]},"public_addr":"172.21.15.46:6800/264199318","cluster_addr":"172.21.15.46:6801/264199318","heartbeat_back_addr":"172.21.15.46:6803/264199318","heartbeat_front_addr":"172.21.15.46:6802/264199318","state":["exists","up"]},{"osd":5,"uuid":"dabb8542-5498-4389-ad9e-0c0a54387723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":3390152579}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6805","nonce":3390152579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6807","nonce":3390152579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6806","nonce":3390152579}]},"public_addr":"172.21.15.46:6804/3390152579","cluster_addr":"172.21.15.46:6805/3390152579","heartbeat_back_addr":"172.21.15.46:6807/3390152579","heartbeat_front_addr":"172.21.15.46:6806/3390152579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.249472+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.780224+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.456157+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.830776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.656408+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:04:46.995384+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":1,"length":4}]},{"pool":14,"snaps":[{"begin":3,"length":4}]},{"pool":33,"snaps":[{"begin":1,"length":1}]},{"pool":64,"snaps":[{"begin":1,"length":1}]},{"pool":94,"snaps":[{"begin":7,"length":2}]},{"pool":104,"snaps":[{"begin":1,"length":1}]},{"pool":122,"snaps":[{"begin":1,"length":2}]},{"pool":140,"snaps":[{"begin":3,"length":1}]},{"pool":146,"snaps":[{"begin":1,"length":4}]},{"pool":152,"snaps":[{"begin":2,"length":1}]},{"pool":153,"snaps":[{"begin":2,"length":2}]},{"pool":163,"snaps":[{"begin":2,"length":3}]},{"pool":169,"snaps":[{"begin":3,"length":4}]},{"pool":172,"snaps":[{"begin":2,"length":4}]},{"pool":174,"snaps":[{"begin":2,"length":2}]},{"pool":184,"snaps":[{"begin":2,"length":1}]},{"pool":185,"snaps":[{"begin":2,"length":3}]},{"pool":186,"snaps":[{"begin":3,"length":1}]},{"pool":196,"snaps":[{"begin":7,"length":1}]},{"pool":199,"snaps":[{"begin":6,"length":2}]},{"pool":225,"snaps":[{"begin":8,"length":1}]},{"pool":236,"snaps":[{"begin":10,"length":1}]},{"pool":242,"snaps":[{"begin":11,"length":2}]},{"pool":255,"snaps":[{"begin":14,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:23:53.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 10 librados: watch_flush enter 2024-09-16T03:23:53.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 10 librados: watch_flush exit 2024-09-16T03:23:53.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 20 client.184154.objecter shutdown clearing up homeless session... 2024-09-16T03:23:53.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 10 client.184154.objecter successfully canceled tick 2024-09-16T03:23:53.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 >> v1:172.21.15.46:6812/149360280 conn(0x7fed7403e260 legacy=0x7fed74040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 10 monclient: shutdown 2024-09-16T03:23:53.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:53.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 >> v1:172.21.15.45:6790/0 conn(0x7fed88137d20 legacy=0x7fed880a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:53.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 shutdown_connections 2024-09-16T03:23:53.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 >> 172.21.15.45:0/4217136811 conn(0x7fed880aa110 msgr2=0x7fed880a06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:53.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 shutdown_connections 2024-09-16T03:23:53.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 1 -- 172.21.15.45:0/4217136811 wait complete. 2024-09-16T03:23:53.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:53.739+0000 7fed8dfbd640 1 librados: shutdown 2024-09-16T03:23:54.757 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T03:23:54.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:23:54.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient: get_monmap_and_config 2024-09-16T03:23:54.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient: build_initial_monmap 2024-09-16T03:23:54.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient: monmap: 2024-09-16T03:23:54.900 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:54.900 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:54.900 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:54.902371+0000 2024-09-16T03:23:54.900 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:54.902371+0000 2024-09-16T03:23:54.901 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:54.901 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:54.901 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:54.901 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:54.901 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:54.901 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:54.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 1 Processor -- start 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 1 -- start start 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient: init 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient(hunting): picked mon.noname-b con 0x7f8dec135ab0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient(hunting): picked mon.noname-a con 0x7f8dec09dee0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient(hunting): picked mon.noname-c con 0x7f8dec13c030 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dec09eee0 con 0x7f8dec09dee0 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dec09f1c0 con 0x7f8dec13c030 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dec09f4a0 con 0x7f8dec135ab0 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:54.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df3a0a640 10 monclient(hunting): authenticate will time out at 2125.551704s 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df0fa5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8dec09dee0 0x7f8dec13a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60294/0 (socket says 172.21.15.45:60294) 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df17a6640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8dec135ab0 0x7f8dec135e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58490/0 (socket says 172.21.15.45:58490) 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.899+0000 7f8df0fa5640 1 -- 172.21.15.45:0/2484719142 learned_addr learned my addr 172.21.15.45:0/2484719142 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1613692308 0 0) 0x7f8dec09eee0 con 0x7f8dec09dee0 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient(hunting): my global_id is 174258 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd8003530 con 0x7f8dec09dee0 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3123973867 0 0) 0x7f8dec09f4a0 con 0x7f8dec135ab0 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient(hunting): my global_id is 184157 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dec09eee0 con 0x7f8dec135ab0 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1039523874 0 0) 0x7f8dec09eee0 con 0x7f8dec135ab0 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 >> v1:172.21.15.46:6789/0 conn(0x7f8dec13c030 legacy=0x7f8dec13e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 >> v1:172.21.15.45:6789/0 conn(0x7f8dec09dee0 legacy=0x7f8dec13a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 monclient: found mon.noname-b 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec09dc30 con 0x7f8dec135ab0 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: _finish_auth 0 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:54.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8de80015b0 con 0x7f8dec135ab0 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 5 monclient: authenticate success, global_id 184157 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: dump: 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8de8001750 con 0x7f8dec135ab0 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 4 set_mon_vals no callback set 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 1 -- 172.21.15.45:0/2484719142 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8de8001b60 con 0x7f8dec135ab0 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de3fff640 10 monclient: dump: 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: shutdown 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2484719142 >> v1:172.21.15.45:6790/0 conn(0x7f8dec135ab0 legacy=0x7f8dec135e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2484719142 shutdown_connections 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2484719142 >> 172.21.15.45:0/2484719142 conn(0x7f8dec0a65a0 msgr2=0x7f8dec0a29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2484719142 shutdown_connections 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2484719142 wait complete. 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: build_initial_monmap 2024-09-16T03:23:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: monmap: 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 librados: starting msgr at 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 librados: starting objecter 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 Processor -- start 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- start start 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 librados: setting wanted keys 2024-09-16T03:23:54.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 librados: calling monclient init 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: init 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient(hunting): picked mon.noname-c con 0x7f8dec09dee0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient(hunting): picked mon.noname-a con 0x7f8dec135ab0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient(hunting): picked mon.noname-b con 0x7f8dec13c030 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dec06cf80 con 0x7f8dec135ab0 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dec138410 con 0x7f8dec13c030 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dec073450 con 0x7f8dec09dee0 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient(hunting): authenticate will time out at 2125.554884s 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df17a6640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8dec09dee0 0x7f8dec072630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58494/0 (socket says 172.21.15.45:58494) 2024-09-16T03:23:54.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df17a6640 1 -- 172.21.15.45:0/2789631294 learned_addr learned my addr 172.21.15.45:0/2789631294 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 client.?.objecter ms_handle_connect 0x7f8dec09dee0 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 client.?.objecter ms_handle_connect 0x7f8dec135ab0 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df17a6640 1 -- 172.21.15.45:0/2789631294 >> v1:172.21.15.45:6790/0 conn(0x7f8dec09dee0 legacy=0x7f8dec072630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df17a6640 1 -- 172.21.15.45:0/2789631294 >> v1:172.21.15.45:6790/0 conn(0x7f8dec09dee0 legacy=0x7f8dec072630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df17a6640 1 --1- 172.21.15.45:0/2789631294 >> v1:172.21.15.45:6790/0 conn(0x7f8dec09dee0 0x7f8dec072630 :-1 s=OPENED pgs=9283 cs=1 l=1).handle_message read tag failed 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df17a6640 1 --1- 172.21.15.45:0/2789631294 >> v1:172.21.15.45:6790/0 conn(0x7f8dec09dee0 0x7f8dec072630 :-1 s=OPENED pgs=9283 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.45:6790/0 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1897163243 0 0) 0x7f8ddc001080 con 0x7f8dec135ab0 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): my global_id is 174261 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd0004ec0 con 0x7f8dec135ab0 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 client.?.objecter ms_handle_connect 0x7f8dec13c030 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1045026139 0 0) 0x7f8dd0004ec0 con 0x7f8dec135ab0 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dd0005350 con 0x7f8dec135ab0 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8ddc0019c0 con 0x7f8dec135ab0 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): dump: 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 794669346 0 0) 0x7f8dec138410 con 0x7f8dec13c030 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient(hunting): my global_id is 164341 2024-09-16T03:23:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd0004ec0 con 0x7f8dec13c030 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 216728562 0 0) 0x7f8dd0005350 con 0x7f8dec135ab0 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 >> v1:172.21.15.45:6790/0 conn(0x7f8dec09dee0 legacy=0x7f8dec072630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 >> v1:172.21.15.46:6789/0 conn(0x7f8dec13c030 legacy=0x7f8dec1d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 monclient: found mon.a 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec073fe0 con 0x7f8dec135ab0 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 5 monclient: authenticate success, global_id 174261 2024-09-16T03:23:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: _renew_subs 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8dec138130 con 0x7f8dec135ab0 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 client.174261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: _renew_subs 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8dec136c10 con 0x7f8dec135ab0 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8df3a0a640 1 librados: init done 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ddc001ce0 con 0x7f8dec135ab0 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f8ddc004c40 con 0x7f8dec135ab0 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de37fe640 4 set_mon_vals no callback set 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.903+0000 7f8de1ffb640 10 monclient: dump: 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8df3a0a640 10 client.174261.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8df3a0a640 15 client.174261.objecter _session_command_op_assign -1 1 2024-09-16T03:23:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8df3a0a640 20 client.174261.objecter _get_session osd=-1 returning homeless 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8df3a0a640 10 client.174261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8df3a0a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 --> v1:172.21.15.45:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8dcc002920 con 0x7f8dec135ab0 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f8ddc010cb0 con 0x7f8dec135ab0 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f8ddc04b4e0 con 0x7f8dec135ab0 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 10 client.174261.objecter ms_dispatch 0x7f8dec000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 3 client.174261.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 3 client.174261.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 10 client.174261.objecter checking command 1 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 20 client.174261.objecter _get_session s=0x7f8dd00461c0 osd=0 3 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 20 client.174261.objecter put_session s=0x7f8dd00461c0 osd=0 3 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 15 client.174261.objecter _session_command_op_remove -1 1 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 20 client.174261.objecter _get_session s=0x7f8dd00461c0 osd=0 3 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 20 client.174261.objecter get_session s=0x7f8dd00461c0 osd=0 3 2024-09-16T03:23:54.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 15 client.174261.objecter _session_command_op_assign 0 1 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 20 client.174261.objecter put_session s=0x7f8dd00461c0 osd=0 4 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 10 client.174261.objecter _send_command 1 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 --> v1:172.21.15.45:6800/3131549395 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8dd004a580 con 0x7f8dd0046870 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 20 client.174261.objecter dump_active .. 0 homeless 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 1851051592 0 0) 0x7f8ddc04b870 con 0x7f8dec135ab0 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8df1fa7640 10 monclient: get_auth_request con 0x7f8dd0046870 auth_method 0 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8df17a6640 10 monclient: get_auth_request con 0x7f8dd003e270 auth_method 0 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 10 client.174261.objecter ms_handle_connect 0x7f8dd0046870 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 10 client.174261.objecter ms_handle_connect 0x7f8dd003e270 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== osd.0 v1:172.21.15.45:6800/3131549395 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8dd004a580 con 0x7f8dd0046870 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 10 client.174261.objecter ms_dispatch 0x7f8dec000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 10 client.174261.objecter _finish_command 1 = system:0 2024-09-16T03:23:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 20 client.174261.objecter put_session s=0x7f8dd00461c0 osd=0 4 2024-09-16T03:23:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.907+0000 7f8de1ffb640 15 client.174261.objecter _session_command_op_remove 0 1 2024-09-16T03:23:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 10 client.174261.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T03:23:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 15 client.174261.objecter _session_command_op_assign -1 2 2024-09-16T03:23:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 20 client.174261.objecter _get_session s=0x7f8dd00461c0 osd=0 3 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 20 client.174261.objecter put_session s=0x7f8dd00461c0 osd=0 3 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 20 client.174261.objecter _get_session s=0x7f8dd00461c0 osd=0 3 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 15 client.174261.objecter _session_command_op_remove -1 2 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 20 client.174261.objecter get_session s=0x7f8dd00461c0 osd=0 3 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 15 client.174261.objecter _session_command_op_assign 0 2 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 20 client.174261.objecter put_session s=0x7f8dd00461c0 osd=0 4 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 10 client.174261.objecter _send_command 2 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 --> v1:172.21.15.45:6800/3131549395 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8dcc0012c0 con 0x7f8dd0046870 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8de1ffb640 1 -- 172.21.15.45:0/2789631294 <== osd.0 v1:172.21.15.45:6800/3131549395 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f8dcc0012c0 con 0x7f8dd0046870 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8de1ffb640 10 client.174261.objecter ms_dispatch 0x7f8dec000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8de1ffb640 10 client.174261.objecter _finish_command 2 = system:0 2024-09-16T03:23:54.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8de1ffb640 20 client.174261.objecter put_session s=0x7f8dd00461c0 osd=0 4 2024-09-16T03:23:54.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.919+0000 7f8de1ffb640 15 client.174261.objecter _session_command_op_remove 0 2 2024-09-16T03:23:54.927 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:23:54.927 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:23:54.927 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:23:54.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 10 librados: watch_flush enter 2024-09-16T03:23:54.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 10 librados: watch_flush exit 2024-09-16T03:23:54.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 10 client.174261.objecter close_session for osd.0 2024-09-16T03:23:54.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 >> v1:172.21.15.45:6800/3131549395 conn(0x7f8dd0046870 legacy=0x7f8dd0048d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 20 client.174261.objecter put_session s=0x7f8dd00461c0 osd=0 1 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 20 client.174261.objecter shutdown clearing up homeless session... 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 10 client.174261.objecter successfully canceled tick 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 >> v1:172.21.15.46:6812/149360280 conn(0x7f8dd003e270 legacy=0x7f8dd0040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 10 monclient: shutdown 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 >> v1:172.21.15.45:6789/0 conn(0x7f8dec135ab0 legacy=0x7f8dec072d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df1fa7640 1 -- 172.21.15.45:0/2789631294 reap_dead start 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 shutdown_connections 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 >> 172.21.15.45:0/2789631294 conn(0x7f8dec0a65a0 msgr2=0x7f8dec0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 shutdown_connections 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 1 -- 172.21.15.45:0/2789631294 wait complete. 2024-09-16T03:23:54.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:54.923+0000 7f8df3a0a640 1 librados: shutdown 2024-09-16T03:23:54.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T03:23:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient: get_monmap_and_config 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient: build_initial_monmap 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient: monmap: 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:55.056843+0000 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:55.056843+0000 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:55.055 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 1 Processor -- start 2024-09-16T03:23:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 1 -- start start 2024-09-16T03:23:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient: init 2024-09-16T03:23:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc8137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc8138100 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc813e2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc80a4f90 con 0x7f4cc813e2a0 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc80a4cb0 con 0x7f4cc8138100 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc809dc70 con 0x7f4cc8137d20 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient(hunting): authenticate will time out at 2125.706295s 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc759e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4cc813e2a0 0x7f4cc8140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60302/0 (socket says 172.21.15.45:60302) 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc759e640 1 -- 172.21.15.45:0/2601471472 learned_addr learned my addr 172.21.15.45:0/2601471472 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc6d9d640 1 --1- 172.21.15.45:0/2601471472 >> v1:172.21.15.45:6790/0 conn(0x7f4cc8137d20 0x7f4cc80a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58506/0 (socket says 172.21.15.45:58506) 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3657154991 0 0) 0x7f4cc80a4f90 con 0x7f4cc813e2a0 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient(hunting): my global_id is 174264 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cb4003530 con 0x7f4cc813e2a0 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3939257686 0 0) 0x7f4cc809dc70 con 0x7f4cc8137d20 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient(hunting): my global_id is 184160 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cc80a4f90 con 0x7f4cc8137d20 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2091840759 0 0) 0x7f4cb4003530 con 0x7f4cc813e2a0 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 >> v1:172.21.15.45:6790/0 conn(0x7f4cc8137d20 legacy=0x7f4cc80a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 >> v1:172.21.15.46:6789/0 conn(0x7f4cc8138100 legacy=0x7f4cc813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: _finish_hunting 0 2024-09-16T03:23:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 monclient: found mon.noname-a 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc809cc50 con 0x7f4cc813e2a0 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: _finish_auth 0 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: _check_auth_tickets 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 5 monclient: authenticate success, global_id 174264 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4cc0001a20 con 0x7f4cc813e2a0 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: dump: 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4cc0001c40 con 0x7f4cc813e2a0 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 1 -- 172.21.15.45:0/2601471472 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4cc00020d0 con 0x7f4cc813e2a0 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4cc659c640 10 monclient: dump: 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.067 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 4 set_mon_vals no callback set 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 10 monclient: shutdown 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 1 -- 172.21.15.45:0/2601471472 >> v1:172.21.15.45:6789/0 conn(0x7f4cc813e2a0 legacy=0x7f4cc8140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 1 -- 172.21.15.45:0/2601471472 shutdown_connections 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 1 -- 172.21.15.45:0/2601471472 >> 172.21.15.45:0/2601471472 conn(0x7f4cc80aa110 msgr2=0x7f4cc80aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.055+0000 7f4ccd147640 1 -- 172.21.15.45:0/2601471472 shutdown_connections 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 -- 172.21.15.45:0/2601471472 wait complete. 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: build_initial_monmap 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: monmap: 2024-09-16T03:23:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 librados: starting msgr at 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 librados: starting objecter 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 Processor -- start 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 -- start start 2024-09-16T03:23:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 librados: setting wanted keys 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 librados: calling monclient init 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: init 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc8137d20 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc8138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc813e2a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc8141aa0 con 0x7f4cc8138100 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc80a3fa0 con 0x7f4cc8137d20 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc8138c30 con 0x7f4cc813e2a0 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient(hunting): authenticate will time out at 2125.709546s 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4cc8138100 0x7f4cc81d1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60314/0 (socket says 172.21.15.45:60314) 2024-09-16T03:23:55.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbffff640 1 -- 172.21.15.45:0/3681825279 learned_addr learned my addr 172.21.15.45:0/3681825279 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 client.?.objecter ms_handle_connect 0x7f4cc813e2a0 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 client.?.objecter ms_handle_connect 0x7f4cc8138100 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 994097865 0 0) 0x7f4cc8141aa0 con 0x7f4cc8138100 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): my global_id is 174267 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ca8004ec0 con 0x7f4cc8138100 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1216414619 0 0) 0x7f4cc8138c30 con 0x7f4cc813e2a0 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): my global_id is 184163 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cc8141aa0 con 0x7f4cc813e2a0 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 client.?.objecter ms_handle_connect 0x7f4cc8137d20 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4157762981 0 0) 0x7f4ca8004ec0 con 0x7f4cc8138100 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cc8138c30 con 0x7f4cc8138100 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4cb0001a40 con 0x7f4cc8138100 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): dump: 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1196115909 0 0) 0x7f4cc8141aa0 con 0x7f4cc813e2a0 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ca8004ec0 con 0x7f4cc813e2a0 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2609668195 0 0) 0x7f4cc80a3fa0 con 0x7f4cc8137d20 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): my global_id is 164347 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cc8141aa0 con 0x7f4cc8137d20 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4cc0001df0 con 0x7f4cc813e2a0 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient(hunting): dump: 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.074 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1167177493 0 0) 0x7f4cc8138c30 con 0x7f4cc8138100 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 >> v1:172.21.15.45:6790/0 conn(0x7f4cc813e2a0 legacy=0x7f4cc81d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 >> v1:172.21.15.46:6789/0 conn(0x7f4cc8137d20 legacy=0x7f4cc813cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 monclient: found mon.a 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc8137240 con 0x7f4cc8138100 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient: _finish_auth 0 2024-09-16T03:23:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 5 monclient: authenticate success, global_id 174267 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: _renew_subs 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc80a3c70 con 0x7f4cc8138100 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 client.174267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: _renew_subs 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4cc813d5b0 con 0x7f4cc8138100 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 librados: init done 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4cb0001d40 con 0x7f4cc8138100 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4cb0003a70 con 0x7f4cc8138100 2024-09-16T03:23:55.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cc559a640 4 set_mon_vals no callback set 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 monclient: dump: 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cac005090 con 0x7f4cc8138100 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4cb0003c10 con 0x7f4cc8138100 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f4cb004b5e0 con 0x7f4cc8138100 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.059+0000 7f4cbf7fe640 10 client.174267.objecter ms_dispatch 0x7f4cc8000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.063+0000 7f4cbf7fe640 3 client.174267.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.063+0000 7f4cbf7fe640 3 client.174267.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.063+0000 7f4cbf7fe640 20 client.174267.objecter dump_active .. 0 homeless 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.063+0000 7f4cc6d9d640 10 monclient: get_auth_request con 0x7f4ca803e2d0 auth_method 0 2024-09-16T03:23:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.063+0000 7f4cbf7fe640 10 client.174267.objecter ms_handle_connect 0x7f4ca803e2d0 2024-09-16T03:23:55.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.067+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4cb0016ae0 con 0x7f4cc8138100 2024-09-16T03:23:55.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.067+0000 7f4cbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:55.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.067+0000 7f4cbf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:55.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.235+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f4cac000ba0 con 0x7f4ca803e2d0 2024-09-16T03:23:55.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.235+0000 7f4cbf7fe640 1 -- 172.21.15.45:0/3681825279 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7f4cac000ba0 con 0x7f4ca803e2d0 2024-09-16T03:23:55.237 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 10 librados: watch_flush enter 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 10 librados: watch_flush exit 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 20 client.174267.objecter shutdown clearing up homeless session... 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 10 client.174267.objecter successfully canceled tick 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 >> v1:172.21.15.46:6812/149360280 conn(0x7f4ca803e2d0 legacy=0x7f4ca8040790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 10 monclient: shutdown 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 >> v1:172.21.15.45:6789/0 conn(0x7f4cc8138100 legacy=0x7f4cc81d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 shutdown_connections 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 >> 172.21.15.45:0/3681825279 conn(0x7f4cc80aa110 msgr2=0x7f4cc8130db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:55.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 shutdown_connections 2024-09-16T03:23:55.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 1 -- 172.21.15.45:0/3681825279 wait complete. 2024-09-16T03:23:55.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.239+0000 7f4ccd147640 1 librados: shutdown 2024-09-16T03:23:55.255 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T03:23:55.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:23:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient: get_monmap_and_config 2024-09-16T03:23:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient: build_initial_monmap 2024-09-16T03:23:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient: monmap: 2024-09-16T03:23:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:55.386231+0000 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:55.386231+0000 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 1 Processor -- start 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 1 -- start start 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient: init 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:55.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient(hunting): picked mon.noname-a con 0x7ff57c074550 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient(hunting): picked mon.noname-b con 0x7ff57c130bc0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient(hunting): picked mon.noname-c con 0x7ff57c13a6f0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c131030 con 0x7ff57c074550 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c09f790 con 0x7ff57c13a6f0 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c09fa70 con 0x7ff57c130bc0 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff581d8c640 10 monclient(hunting): authenticate will time out at 2126.035777s 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff57affd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff57c130bc0 0x7ff57c138fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58512/0 (socket says 172.21.15.45:58512) 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff57affd640 1 -- 172.21.15.45:0/3669628996 learned_addr learned my addr 172.21.15.45:0/3669628996 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff57b7fe640 1 --1- 172.21.15.45:0/3669628996 >> v1:172.21.15.45:6789/0 conn(0x7ff57c074550 0x7ff57c074930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60330/0 (socket says 172.21.15.45:60330) 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff57affd640 1 -- 172.21.15.45:0/3669628996 >> v1:172.21.15.45:6790/0 conn(0x7ff57c130bc0 legacy=0x7ff57c138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff57affd640 1 -- 172.21.15.45:0/3669628996 >> v1:172.21.15.45:6790/0 conn(0x7ff57c130bc0 legacy=0x7ff57c138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:23:55.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff57affd640 1 --1- 172.21.15.45:0/3669628996 >> v1:172.21.15.45:6790/0 conn(0x7ff57c130bc0 0x7ff57c138fd0 :-1 s=OPENED pgs=9286 cs=1 l=1).handle_message read tag failed 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff57affd640 1 --1- 172.21.15.45:0/3669628996 >> v1:172.21.15.45:6790/0 conn(0x7ff57c130bc0 0x7ff57c138fd0 :-1 s=OPENED pgs=9286 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.383+0000 7ff57a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.45:6790/0 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 -- 172.21.15.45:0/3669628996 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 989924456 0 0) 0x7ff57c131030 con 0x7ff57c074550 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient(hunting): my global_id is 174270 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 -- 172.21.15.45:0/3669628996 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff564004640 con 0x7ff57c074550 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 -- 172.21.15.45:0/3669628996 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3659407607 0 0) 0x7ff564004640 con 0x7ff57c074550 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 -- 172.21.15.45:0/3669628996 >> v1:172.21.15.45:6790/0 conn(0x7ff57c130bc0 legacy=0x7ff57c138fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 -- 172.21.15.45:0/3669628996 >> v1:172.21.15.46:6789/0 conn(0x7ff57c13a6f0 legacy=0x7ff57c13cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 monclient: found mon.noname-a 2024-09-16T03:23:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 -- 172.21.15.45:0/3669628996 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c09fd50 con 0x7ff57c074550 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 -- 172.21.15.45:0/3669628996 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff56c0016a0 con 0x7ff57c074550 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: dump: 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 5 monclient: authenticate success, global_id 174270 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 -- 172.21.15.45:0/3669628996 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff56c0018c0 con 0x7ff57c074550 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 1 -- 172.21.15.45:0/3669628996 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff56c001cd0 con 0x7ff57c074550 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57a7fc640 10 monclient: dump: 2024-09-16T03:23:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 4 set_mon_vals no callback set 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient: shutdown 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 -- 172.21.15.45:0/3669628996 >> v1:172.21.15.45:6789/0 conn(0x7ff57c074550 legacy=0x7ff57c074930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 -- 172.21.15.45:0/3669628996 shutdown_connections 2024-09-16T03:23:55.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 -- 172.21.15.45:0/3669628996 >> 172.21.15.45:0/3669628996 conn(0x7ff57c0a65a0 msgr2=0x7ff57c0a29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:55.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 -- 172.21.15.45:0/3669628996 shutdown_connections 2024-09-16T03:23:55.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 -- 172.21.15.45:0/3669628996 wait complete. 2024-09-16T03:23:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient: build_initial_monmap 2024-09-16T03:23:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient: monmap: 2024-09-16T03:23:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 librados: starting msgr at 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 librados: starting objecter 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 Processor -- start 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 -- start start 2024-09-16T03:23:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 librados: setting wanted keys 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 librados: calling monclient init 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient: init 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient(hunting): picked mon.noname-a con 0x7ff57c074550 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient(hunting): picked mon.noname-c con 0x7ff57c130bc0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient(hunting): picked mon.noname-b con 0x7ff57c13a6f0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c133a20 con 0x7ff57c074550 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c0a4d70 con 0x7ff57c13a6f0 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff57c074550 0x7ff57c138f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60338/0 (socket says 172.21.15.45:60338) 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff57b7fe640 1 -- 172.21.15.45:0/3274346639 learned_addr learned my addr 172.21.15.45:0/3274346639 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c133740 con 0x7ff57c130bc0 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff581d8c640 10 monclient(hunting): authenticate will time out at 2126.039653s 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 client.?.objecter ms_handle_connect 0x7ff57c074550 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 client.?.objecter ms_handle_connect 0x7ff57c130bc0 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 508251391 0 0) 0x7ff57c133a20 con 0x7ff57c074550 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 monclient(hunting): my global_id is 174273 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff568004ec0 con 0x7ff57c074550 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3013957790 0 0) 0x7ff57c133740 con 0x7ff57c130bc0 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 10 monclient(hunting): my global_id is 184166 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.387+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff57c133a20 con 0x7ff57c130bc0 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 client.?.objecter ms_handle_connect 0x7ff57c13a6f0 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3281357571 0 0) 0x7ff568004ec0 con 0x7ff57c074550 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff57c133740 con 0x7ff57c074550 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff56c001c10 con 0x7ff57c074550 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): dump: 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3012301109 0 0) 0x7ff57c133a20 con 0x7ff57c130bc0 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff568004ec0 con 0x7ff57c130bc0 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff5700018f0 con 0x7ff57c130bc0 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): dump: 2024-09-16T03:23:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1288598595 0 0) 0x7ff57c0a4d70 con 0x7ff57c13a6f0 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient(hunting): my global_id is 164353 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff57c133a20 con 0x7ff57c13a6f0 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3597248995 0 0) 0x7ff57c133740 con 0x7ff57c074550 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 >> v1:172.21.15.45:6790/0 conn(0x7ff57c130bc0 legacy=0x7ff57c1cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 >> v1:172.21.15.46:6789/0 conn(0x7ff57c13a6f0 legacy=0x7ff57c1d0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient: _finish_hunting 0 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 monclient: found mon.a 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c1396a0 con 0x7ff57c074550 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient: _finish_auth 0 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient: _check_auth_tickets 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:55.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 5 monclient: authenticate success, global_id 174273 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 10 monclient: _renew_subs 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff57c132220 con 0x7ff57c074550 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 10 client.174273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 10 monclient: _renew_subs 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff57c1c4b70 con 0x7ff57c074550 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 1 librados: init done 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff56c001820 con 0x7ff57c074550 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7ff56c004b40 con 0x7ff57c074550 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:55.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff5797fa640 4 set_mon_vals no callback set 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient: dump: 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 10 client.174273.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 15 client.174273.objecter _session_command_op_assign -1 1 2024-09-16T03:23:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 20 client.174273.objecter _get_session osd=-1 returning homeless 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 10 client.174273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7ff55c002920 con 0x7ff57c074550 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7ff56c010bb0 con 0x7ff57c074550 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7ff56c04b4a0 con 0x7ff57c074550 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 client.174273.objecter ms_dispatch 0x7ff57c000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 3 client.174273.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 3 client.174273.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 client.174273.objecter checking command 1 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 20 client.174273.objecter _get_session s=0x7ff5680482c0 osd=1 3 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 20 client.174273.objecter put_session s=0x7ff5680482c0 osd=1 3 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 15 client.174273.objecter _session_command_op_remove -1 1 2024-09-16T03:23:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 20 client.174273.objecter _get_session s=0x7ff5680482c0 osd=1 3 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 20 client.174273.objecter get_session s=0x7ff5680482c0 osd=1 3 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 15 client.174273.objecter _session_command_op_assign 1 1 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 20 client.174273.objecter put_session s=0x7ff5680482c0 osd=1 4 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 client.174273.objecter _send_command 1 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6808/3228817795 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff56804c680 con 0x7ff568048970 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 20 client.174273.objecter dump_active .. 0 homeless 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 1851051592 0 0) 0x7ff56c04b8f0 con 0x7ff57c074550 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff57affd640 10 monclient: get_auth_request con 0x7ff5680404c0 auth_method 0 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff57bfff640 10 monclient: get_auth_request con 0x7ff568048970 auth_method 0 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 client.174273.objecter ms_handle_connect 0x7ff568048970 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 client.174273.objecter ms_handle_connect 0x7ff5680404c0 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== osd.1 v1:172.21.15.45:6808/3228817795 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff56804c680 con 0x7ff568048970 2024-09-16T03:23:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 client.174273.objecter ms_dispatch 0x7ff57c000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:23:55.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 10 client.174273.objecter _finish_command 1 = system:0 2024-09-16T03:23:55.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 20 client.174273.objecter put_session s=0x7ff5680482c0 osd=1 4 2024-09-16T03:23:55.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.391+0000 7ff563fff640 15 client.174273.objecter _session_command_op_remove 1 1 2024-09-16T03:23:55.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 10 client.174273.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T03:23:55.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 15 client.174273.objecter _session_command_op_assign -1 2 2024-09-16T03:23:55.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 20 client.174273.objecter _get_session s=0x7ff5680482c0 osd=1 3 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 20 client.174273.objecter put_session s=0x7ff5680482c0 osd=1 3 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 20 client.174273.objecter _get_session s=0x7ff5680482c0 osd=1 3 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 15 client.174273.objecter _session_command_op_remove -1 2 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 20 client.174273.objecter get_session s=0x7ff5680482c0 osd=1 3 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 15 client.174273.objecter _session_command_op_assign 1 2 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 20 client.174273.objecter put_session s=0x7ff5680482c0 osd=1 4 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 10 client.174273.objecter _send_command 2 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 --> v1:172.21.15.45:6808/3228817795 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff55c0012c0 con 0x7ff568048970 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff563fff640 1 -- 172.21.15.45:0/3274346639 <== osd.1 v1:172.21.15.45:6808/3228817795 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7ff55c0012c0 con 0x7ff568048970 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff563fff640 10 client.174273.objecter ms_dispatch 0x7ff57c000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff563fff640 10 client.174273.objecter _finish_command 2 = system:0 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff563fff640 20 client.174273.objecter put_session s=0x7ff5680482c0 osd=1 4 2024-09-16T03:23:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.403+0000 7ff563fff640 15 client.174273.objecter _session_command_op_remove 1 2 2024-09-16T03:23:55.413 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:23:55.413 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:23:55.413 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 10 librados: watch_flush enter 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 10 librados: watch_flush exit 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 10 client.174273.objecter close_session for osd.1 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 >> v1:172.21.15.45:6808/3228817795 conn(0x7ff568048970 legacy=0x7ff56804ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 20 client.174273.objecter put_session s=0x7ff5680482c0 osd=1 1 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 20 client.174273.objecter shutdown clearing up homeless session... 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 10 client.174273.objecter successfully canceled tick 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 >> v1:172.21.15.46:6812/149360280 conn(0x7ff5680404c0 legacy=0x7ff568042980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 10 monclient: shutdown 2024-09-16T03:23:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 >> v1:172.21.15.45:6789/0 conn(0x7ff57c074550 legacy=0x7ff57c138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff57bfff640 1 -- 172.21.15.45:0/3274346639 reap_dead start 2024-09-16T03:23:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 shutdown_connections 2024-09-16T03:23:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 >> 172.21.15.45:0/3274346639 conn(0x7ff57c0a65a0 msgr2=0x7ff57c12fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 shutdown_connections 2024-09-16T03:23:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 1 -- 172.21.15.45:0/3274346639 wait complete. 2024-09-16T03:23:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.407+0000 7ff581d8c640 1 librados: shutdown 2024-09-16T03:23:55.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T03:23:55.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.531+0000 7f697060d640 10 monclient: get_monmap_and_config 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.531+0000 7f697060d640 10 monclient: build_initial_monmap 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.531+0000 7f697060d640 10 monclient: monmap: 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:55.536132+0000 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:55.536132+0000 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:55.535 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 Processor -- start 2024-09-16T03:23:55.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 -- start start 2024-09-16T03:23:55.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient: init 2024-09-16T03:23:55.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient(hunting): picked mon.noname-b con 0x7f69680a21c0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient(hunting): picked mon.noname-a con 0x7f69680a2cb0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient(hunting): picked mon.noname-c con 0x7f696813ab20 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696809ed90 con 0x7f69680a2cb0 2024-09-16T03:23:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696809e4f0 con 0x7f696813ab20 2024-09-16T03:23:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696809d910 con 0x7f69680a21c0 2024-09-16T03:23:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient(hunting): authenticate will time out at 2126.185700s 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696dba8640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f69680a2cb0 0x7f6968139400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60342/0 (socket says 172.21.15.45:60342) 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696dba8640 1 -- 172.21.15.45:0/1683662837 learned_addr learned my addr 172.21.15.45:0/1683662837 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1026139363 0 0) 0x7f696809ed90 con 0x7f69680a2cb0 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient(hunting): my global_id is 174276 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6950003530 con 0x7f69680a2cb0 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862741895 0 0) 0x7f696809d910 con 0x7f69680a21c0 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient(hunting): my global_id is 184169 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f696809ed90 con 0x7f69680a21c0 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 489278231 0 0) 0x7f6950003530 con 0x7f69680a2cb0 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 >> v1:172.21.15.45:6790/0 conn(0x7f69680a21c0 legacy=0x7f69680a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 >> v1:172.21.15.46:6789/0 conn(0x7f696813ab20 legacy=0x7f696813cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: _finish_hunting 0 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 monclient: found mon.noname-a 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696809d450 con 0x7f69680a2cb0 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: _finish_auth 0 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: _check_auth_tickets 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:55.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f695c0016a0 con 0x7f69680a2cb0 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: dump: 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.541 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 5 monclient: authenticate success, global_id 174276 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f695c001840 con 0x7f69680a2cb0 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 1 -- 172.21.15.45:0/1683662837 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f695c001c10 con 0x7f69680a2cb0 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f696d3a7640 10 monclient: dump: 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 4 set_mon_vals no callback set 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient: shutdown 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 -- 172.21.15.45:0/1683662837 >> v1:172.21.15.45:6789/0 conn(0x7f69680a2cb0 legacy=0x7f6968139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 -- 172.21.15.45:0/1683662837 shutdown_connections 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 -- 172.21.15.45:0/1683662837 >> 172.21.15.45:0/1683662837 conn(0x7f69680a6de0 msgr2=0x7f69680a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 -- 172.21.15.45:0/1683662837 shutdown_connections 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 -- 172.21.15.45:0/1683662837 wait complete. 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient: build_initial_monmap 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 10 monclient: monmap: 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:55.544 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:55.545 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 librados: starting msgr at 2024-09-16T03:23:55.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.535+0000 7f697060d640 1 librados: starting objecter 2024-09-16T03:23:55.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 Processor -- start 2024-09-16T03:23:55.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 -- start start 2024-09-16T03:23:55.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 librados: setting wanted keys 2024-09-16T03:23:55.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 librados: calling monclient init 2024-09-16T03:23:55.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient: init 2024-09-16T03:23:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient(hunting): picked mon.noname-c con 0x7f69680a21c0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient(hunting): picked mon.noname-a con 0x7f69680a2cb0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient(hunting): picked mon.noname-b con 0x7f696813ab20 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6968140ff0 con 0x7f69680a2cb0 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6968140ca0 con 0x7f696813ab20 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6968141f70 con 0x7f69680a21c0 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient(hunting): authenticate will time out at 2126.188650s 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f696e3a9640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f69680a21c0 0x7f6968139c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58532/0 (socket says 172.21.15.45:58532) 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f696dba8640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f69680a2cb0 0x7f69681d1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60352/0 (socket says 172.21.15.45:60352) 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f696e3a9640 1 -- 172.21.15.45:0/2426552283 learned_addr learned my addr 172.21.15.45:0/2426552283 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 client.?.objecter ms_handle_connect 0x7f69680a21c0 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 client.?.objecter ms_handle_connect 0x7f69680a2cb0 2024-09-16T03:23:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1511143555 0 0) 0x7f6968141f70 con 0x7f69680a21c0 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): my global_id is 184172 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6954004ef0 con 0x7f69680a21c0 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776053633 0 0) 0x7f6968140ff0 con 0x7f69680a2cb0 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): my global_id is 174279 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6968141f70 con 0x7f69680a2cb0 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 client.?.objecter ms_handle_connect 0x7f696813ab20 2024-09-16T03:23:55.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3487409050 0 0) 0x7f6968141f70 con 0x7f69680a2cb0 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6968140ff0 con 0x7f69680a2cb0 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f695c001b90 con 0x7f69680a2cb0 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): dump: 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 379061307 0 0) 0x7f6954004ef0 con 0x7f69680a21c0 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6968141f70 con 0x7f69680a21c0 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f69600016a0 con 0x7f69680a21c0 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): dump: 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632178583 0 0) 0x7f6968140ca0 con 0x7f696813ab20 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient(hunting): my global_id is 164359 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6954004ef0 con 0x7f696813ab20 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1732461113 0 0) 0x7f6968140ff0 con 0x7f69680a2cb0 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 >> v1:172.21.15.45:6790/0 conn(0x7f69680a21c0 legacy=0x7f6968139c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 >> v1:172.21.15.46:6789/0 conn(0x7f696813ab20 legacy=0x7f69681d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 monclient: found mon.a 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69680702d0 con 0x7f69680a2cb0 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient: _finish_auth 0 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 5 monclient: authenticate success, global_id 174279 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient: _renew_subs 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69681409c0 con 0x7f69680a2cb0 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 client.174279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient: _renew_subs 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6968141420 con 0x7f69680a2cb0 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 librados: init done 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f695c001820 con 0x7f69680a2cb0 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f695c004c40 con 0x7f69680a2cb0 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f696cba6640 4 set_mon_vals no callback set 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 monclient: dump: 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6948005090 con 0x7f69680a2cb0 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f695c004de0 con 0x7f69680a2cb0 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f695c04b540 con 0x7f69680a2cb0 2024-09-16T03:23:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 client.174279.objecter ms_dispatch 0x7f6968000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 3 client.174279.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 3 client.174279.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 20 client.174279.objecter dump_active .. 0 homeless 2024-09-16T03:23:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f696e3a9640 10 monclient: get_auth_request con 0x7f695403e1d0 auth_method 0 2024-09-16T03:23:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.539+0000 7f6966ffd640 10 client.174279.objecter ms_handle_connect 0x7f695403e1d0 2024-09-16T03:23:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.543+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f695c016ae0 con 0x7f69680a2cb0 2024-09-16T03:23:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.543+0000 7f6966ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.543+0000 7f6966ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:55.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.715+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f6948000ba0 con 0x7f695403e1d0 2024-09-16T03:23:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f6966ffd640 1 -- 172.21.15.45:0/2426552283 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7f6948000ba0 con 0x7f695403e1d0 2024-09-16T03:23:55.718 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T03:23:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 10 librados: watch_flush enter 2024-09-16T03:23:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 10 librados: watch_flush exit 2024-09-16T03:23:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 20 client.174279.objecter shutdown clearing up homeless session... 2024-09-16T03:23:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 10 client.174279.objecter successfully canceled tick 2024-09-16T03:23:55.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 >> v1:172.21.15.46:6812/149360280 conn(0x7f695403e1d0 legacy=0x7f6954040690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 10 monclient: shutdown 2024-09-16T03:23:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 >> v1:172.21.15.45:6789/0 conn(0x7f69680a2cb0 legacy=0x7f69681d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 shutdown_connections 2024-09-16T03:23:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 >> 172.21.15.45:0/2426552283 conn(0x7f69680a6de0 msgr2=0x7f69680a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 shutdown_connections 2024-09-16T03:23:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 1 -- 172.21.15.45:0/2426552283 wait complete. 2024-09-16T03:23:55.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.719+0000 7f697060d640 1 librados: shutdown 2024-09-16T03:23:55.737 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T03:23:55.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:23:55.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient: get_monmap_and_config 2024-09-16T03:23:55.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient: build_initial_monmap 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient: monmap: 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:55.864946+0000 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:55.864946+0000 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:55.863 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 1 Processor -- start 2024-09-16T03:23:55.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 1 -- start start 2024-09-16T03:23:55.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient: init 2024-09-16T03:23:55.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:55.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:55.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient(hunting): picked mon.noname-a con 0x7f6e400a1900 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:55.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient(hunting): picked mon.noname-b con 0x7f6e400a23f0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:55.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient(hunting): picked mon.noname-c con 0x7f6e4013b2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e4009fff0 con 0x7f6e400a1900 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e4009f4b0 con 0x7f6e4013b2a0 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e4006e9c0 con 0x7f6e400a23f0 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient(hunting): authenticate will time out at 2126.514494s 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6e400a1900 0x7f6e400a1ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60358/0 (socket says 172.21.15.45:60358) 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3ffff640 1 -- 172.21.15.45:0/2412969664 learned_addr learned my addr 172.21.15.45:0/2412969664 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3f7fe640 1 --1- 172.21.15.45:0/2412969664 >> v1:172.21.15.45:6790/0 conn(0x7f6e400a23f0 0x7f6e40139b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58538/0 (socket says 172.21.15.45:58538) 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 34526998 0 0) 0x7f6e4006e9c0 con 0x7f6e400a23f0 2024-09-16T03:23:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient(hunting): my global_id is 184175 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e28005740 con 0x7f6e400a23f0 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2192712388 0 0) 0x7f6e4009fff0 con 0x7f6e400a1900 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient(hunting): my global_id is 174282 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e4006e9c0 con 0x7f6e400a1900 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3633832355 0 0) 0x7f6e28005740 con 0x7f6e400a23f0 2024-09-16T03:23:55.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 >> v1:172.21.15.46:6789/0 conn(0x7f6e4013b2a0 legacy=0x7f6e4013d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 >> v1:172.21.15.45:6789/0 conn(0x7f6e400a1900 legacy=0x7f6e400a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: _finish_hunting 0 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 monclient: found mon.noname-b 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e400a3360 con 0x7f6e400a23f0 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: _finish_auth 0 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: _check_auth_tickets 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6e340016a0 con 0x7f6e400a23f0 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: dump: 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 5 monclient: authenticate success, global_id 184175 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e34001840 con 0x7f6e400a23f0 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 4 set_mon_vals no callback set 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 1 -- 172.21.15.45:0/2412969664 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6e34001c90 con 0x7f6e400a23f0 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e3effd640 10 monclient: dump: 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.877 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 10 monclient: shutdown 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 1 -- 172.21.15.45:0/2412969664 >> v1:172.21.15.45:6790/0 conn(0x7f6e400a23f0 legacy=0x7f6e40139b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.863+0000 7f6e46706640 1 -- 172.21.15.45:0/2412969664 shutdown_connections 2024-09-16T03:23:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- 172.21.15.45:0/2412969664 >> 172.21.15.45:0/2412969664 conn(0x7f6e400a65a0 msgr2=0x7f6e400a29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- 172.21.15.45:0/2412969664 shutdown_connections 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- 172.21.15.45:0/2412969664 wait complete. 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: build_initial_monmap 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: monmap: 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.879 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 librados: starting msgr at 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 librados: starting objecter 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 Processor -- start 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- start start 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 librados: setting wanted keys 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 librados: calling monclient init 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: init 2024-09-16T03:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient(hunting): picked mon.noname-c con 0x7f6e400a1900 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient(hunting): picked mon.noname-b con 0x7f6e400a23f0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient(hunting): picked mon.noname-a con 0x7f6e4013b2a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e400a4d40 con 0x7f6e4013b2a0 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e4006cf80 con 0x7f6e400a23f0 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e4013a7e0 con 0x7f6e400a1900 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient(hunting): authenticate will time out at 2126.517690s 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3ffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6e400a1900 0x7f6e401c9600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58550/0 (socket says 172.21.15.45:58550) 2024-09-16T03:23:55.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3ffff640 1 -- 172.21.15.45:0/4037117088 learned_addr learned my addr 172.21.15.45:0/4037117088 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 client.?.objecter ms_handle_connect 0x7f6e400a1900 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 client.?.objecter ms_handle_connect 0x7f6e4013b2a0 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3429583657 0 0) 0x7f6e4013a7e0 con 0x7f6e400a1900 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): my global_id is 184178 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e2c004ef0 con 0x7f6e400a1900 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731919579 0 0) 0x7f6e400a4d40 con 0x7f6e4013b2a0 2024-09-16T03:23:55.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): my global_id is 174285 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e4013a7e0 con 0x7f6e4013b2a0 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 client.?.objecter ms_handle_connect 0x7f6e400a23f0 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4042664376 0 0) 0x7f6e2c004ef0 con 0x7f6e400a1900 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e400a4d40 con 0x7f6e400a1900 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6e300018d0 con 0x7f6e400a1900 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): dump: 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 242449189 0 0) 0x7f6e4013a7e0 con 0x7f6e4013b2a0 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e2c004ef0 con 0x7f6e4013b2a0 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6e38001c00 con 0x7f6e4013b2a0 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): dump: 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4273631945 0 0) 0x7f6e4006cf80 con 0x7f6e400a23f0 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient(hunting): my global_id is 164365 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e4013a7e0 con 0x7f6e400a23f0 2024-09-16T03:23:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2555503669 0 0) 0x7f6e400a4d40 con 0x7f6e400a1900 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 >> v1:172.21.15.46:6789/0 conn(0x7f6e400a23f0 legacy=0x7f6e4012ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 >> v1:172.21.15.45:6789/0 conn(0x7f6e4013b2a0 legacy=0x7f6e401d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient: _finish_hunting 0 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 monclient: found mon.c 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4012f3d0 con 0x7f6e400a1900 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient: _finish_auth 0 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:55.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 5 monclient: authenticate success, global_id 184178 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: _renew_subs 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6e40073780 con 0x7f6e400a1900 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 client.184178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: _renew_subs 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6e401d6560 con 0x7f6e400a1900 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 librados: init done 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e300047f0 con 0x7f6e400a1900 2024-09-16T03:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f6e30004b80 con 0x7f6e400a1900 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3e7fc640 4 set_mon_vals no callback set 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 monclient: dump: 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 client.184178.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 15 client.184178.objecter _session_command_op_assign -1 1 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 20 client.184178.objecter _get_session osd=-1 returning homeless 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 client.184178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6e20002920 con 0x7f6e400a1900 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f6e30004d20 con 0x7f6e400a1900 2024-09-16T03:23:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f6e3004b5c0 con 0x7f6e400a1900 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.867+0000 7f6e3cff9640 10 client.184178.objecter ms_dispatch 0x7f6e40000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 3 client.184178.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 3 client.184178.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 10 client.184178.objecter checking command 1 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 20 client.184178.objecter _get_session s=0x7f6e2c046120 osd=2 3 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 20 client.184178.objecter put_session s=0x7f6e2c046120 osd=2 3 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 15 client.184178.objecter _session_command_op_remove -1 1 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 20 client.184178.objecter _get_session s=0x7f6e2c046120 osd=2 3 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 20 client.184178.objecter get_session s=0x7f6e2c046120 osd=2 3 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 15 client.184178.objecter _session_command_op_assign 2 1 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 20 client.184178.objecter put_session s=0x7f6e2c046120 osd=2 4 2024-09-16T03:23:55.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 10 client.184178.objecter _send_command 1 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6804/207876738 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c04a4e0 con 0x7f6e2c0467d0 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 20 client.184178.objecter dump_active .. 0 homeless 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 1851051592 0 0) 0x7f6e3004b950 con 0x7f6e400a1900 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e44ca3640 10 monclient: get_auth_request con 0x7f6e2c0467d0 auth_method 0 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3f7fe640 10 monclient: get_auth_request con 0x7f6e2c03e220 auth_method 0 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 10 client.184178.objecter ms_handle_connect 0x7f6e2c0467d0 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 10 client.184178.objecter ms_handle_connect 0x7f6e2c03e220 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== osd.2 v1:172.21.15.45:6804/207876738 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6e2c04a4e0 con 0x7f6e2c0467d0 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 10 client.184178.objecter ms_dispatch 0x7f6e40000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 10 client.184178.objecter _finish_command 1 = system:0 2024-09-16T03:23:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 20 client.184178.objecter put_session s=0x7f6e2c046120 osd=2 4 2024-09-16T03:23:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.871+0000 7f6e3cff9640 15 client.184178.objecter _session_command_op_remove 2 1 2024-09-16T03:23:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 10 client.184178.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T03:23:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 15 client.184178.objecter _session_command_op_assign -1 2 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 20 client.184178.objecter _get_session s=0x7f6e2c046120 osd=2 3 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 20 client.184178.objecter put_session s=0x7f6e2c046120 osd=2 3 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 20 client.184178.objecter _get_session s=0x7f6e2c046120 osd=2 3 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 15 client.184178.objecter _session_command_op_remove -1 2 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 20 client.184178.objecter get_session s=0x7f6e2c046120 osd=2 3 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 15 client.184178.objecter _session_command_op_assign 2 2 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 20 client.184178.objecter put_session s=0x7f6e2c046120 osd=2 4 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 10 client.184178.objecter _send_command 2 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 --> v1:172.21.15.45:6804/207876738 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6e200012c0 con 0x7f6e2c0467d0 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e3cff9640 1 -- 172.21.15.45:0/4037117088 <== osd.2 v1:172.21.15.45:6804/207876738 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f6e200012c0 con 0x7f6e2c0467d0 2024-09-16T03:23:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e3cff9640 10 client.184178.objecter ms_dispatch 0x7f6e40000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:23:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e3cff9640 10 client.184178.objecter _finish_command 2 = system:0 2024-09-16T03:23:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e3cff9640 20 client.184178.objecter put_session s=0x7f6e2c046120 osd=2 4 2024-09-16T03:23:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e3cff9640 15 client.184178.objecter _session_command_op_remove 2 2 2024-09-16T03:23:55.895 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:23:55.895 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:23:55.895 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:23:55.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 10 librados: watch_flush enter 2024-09-16T03:23:55.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 10 librados: watch_flush exit 2024-09-16T03:23:55.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 10 client.184178.objecter close_session for osd.2 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 >> v1:172.21.15.45:6804/207876738 conn(0x7f6e2c0467d0 legacy=0x7f6e2c048c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 20 client.184178.objecter put_session s=0x7f6e2c046120 osd=2 1 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 20 client.184178.objecter shutdown clearing up homeless session... 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 10 client.184178.objecter successfully canceled tick 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 >> v1:172.21.15.46:6812/149360280 conn(0x7f6e2c03e220 legacy=0x7f6e2c0406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 10 monclient: shutdown 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 >> v1:172.21.15.45:6790/0 conn(0x7f6e400a1900 legacy=0x7f6e401c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e44ca3640 1 -- 172.21.15.45:0/4037117088 reap_dead start 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 shutdown_connections 2024-09-16T03:23:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 >> 172.21.15.45:0/4037117088 conn(0x7f6e400a65a0 msgr2=0x7f6e40131930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 shutdown_connections 2024-09-16T03:23:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 1 -- 172.21.15.45:0/4037117088 wait complete. 2024-09-16T03:23:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:55.883+0000 7f6e46706640 1 librados: shutdown 2024-09-16T03:23:55.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T03:23:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.011+0000 7fef28350640 10 monclient: get_monmap_and_config 2024-09-16T03:23:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.011+0000 7fef28350640 10 monclient: build_initial_monmap 2024-09-16T03:23:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.011+0000 7fef28350640 10 monclient: monmap: 2024-09-16T03:23:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:56.015292+0000 2024-09-16T03:23:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:56.015292+0000 2024-09-16T03:23:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:56.014 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 Processor -- start 2024-09-16T03:23:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- start start 2024-09-16T03:23:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: init 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient(hunting): picked mon.noname-b con 0x7fef200a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient(hunting): picked mon.noname-c con 0x7fef200a5810 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient(hunting): picked mon.noname-a con 0x7fef20073d40 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef2009d9e0 con 0x7fef20073d40 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef2009dc20 con 0x7fef200a5810 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef20073360 con 0x7fef200a4d20 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient(hunting): authenticate will time out at 2126.664924s 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef260ec640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fef200a4d20 0x7fef200a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58566/0 (socket says 172.21.15.45:58566) 2024-09-16T03:23:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef260ec640 1 -- 172.21.15.45:0/3854451680 learned_addr learned my addr 172.21.15.45:0/3854451680 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099226700 0 0) 0x7fef20073360 con 0x7fef200a4d20 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient(hunting): my global_id is 184181 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef08003530 con 0x7fef200a4d20 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660357186 0 0) 0x7fef2009d9e0 con 0x7fef20073d40 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient(hunting): my global_id is 174288 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef20073360 con 0x7fef20073d40 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1706333685 0 0) 0x7fef08003530 con 0x7fef200a4d20 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 >> v1:172.21.15.46:6789/0 conn(0x7fef200a5810 legacy=0x7fef2013cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 >> v1:172.21.15.45:6789/0 conn(0x7fef20073d40 legacy=0x7fef20140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: _finish_hunting 0 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 monclient: found mon.noname-b 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2009cf30 con 0x7fef200a4d20 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: _finish_auth 0 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: _check_auth_tickets 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 5 monclient: authenticate success, global_id 184181 2024-09-16T03:23:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fef10001a20 con 0x7fef200a4d20 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: dump: 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.026 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef10001bc0 con 0x7fef200a4d20 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 4 set_mon_vals no callback set 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 1 -- 172.21.15.45:0/3854451680 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fef10002010 con 0x7fef200a4d20 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef250ea640 10 monclient: dump: 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.027 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: shutdown 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- 172.21.15.45:0/3854451680 >> v1:172.21.15.45:6790/0 conn(0x7fef200a4d20 legacy=0x7fef200a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- 172.21.15.45:0/3854451680 shutdown_connections 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- 172.21.15.45:0/3854451680 >> 172.21.15.45:0/3854451680 conn(0x7fef200aa720 msgr2=0x7fef200aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- 172.21.15.45:0/3854451680 shutdown_connections 2024-09-16T03:23:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- 172.21.15.45:0/3854451680 wait complete. 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: build_initial_monmap 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: monmap: 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 librados: starting msgr at 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 librados: starting objecter 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 Processor -- start 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 -- start start 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 librados: setting wanted keys 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 1 librados: calling monclient init 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: init 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient(hunting): picked mon.noname-b con 0x7fef20073d40 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.015+0000 7fef28350640 10 monclient(hunting): picked mon.noname-c con 0x7fef200a4d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 monclient(hunting): picked mon.noname-a con 0x7fef200a5810 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef201405e0 con 0x7fef200a5810 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef2013ea00 con 0x7fef20073d40 2024-09-16T03:23:56.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef201408c0 con 0x7fef200a4d20 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 monclient(hunting): authenticate will time out at 2126.668067s 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef268ed640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fef200a5810 0x7fef201d5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60386/0 (socket says 172.21.15.45:60386) 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef268ed640 1 -- 172.21.15.45:0/3139410553 learned_addr learned my addr 172.21.15.45:0/3139410553 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 client.?.objecter ms_handle_connect 0x7fef200a4d20 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 client.?.objecter ms_handle_connect 0x7fef200a5810 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331518236 0 0) 0x7fef201405e0 con 0x7fef200a5810 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): my global_id is 174291 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef0c004ec0 con 0x7fef200a5810 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169515724 0 0) 0x7fef201408c0 con 0x7fef200a4d20 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): my global_id is 184184 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef201405e0 con 0x7fef200a4d20 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 client.?.objecter ms_handle_connect 0x7fef20073d40 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2310826383 0 0) 0x7fef0c004ec0 con 0x7fef200a5810 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef201408c0 con 0x7fef200a5810 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fef18001920 con 0x7fef200a5810 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): dump: 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2540917872 0 0) 0x7fef201405e0 con 0x7fef200a4d20 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef0c004ec0 con 0x7fef200a4d20 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fef14001880 con 0x7fef200a4d20 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): dump: 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3356280338 0 0) 0x7fef2013ea00 con 0x7fef20073d40 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient(hunting): my global_id is 164371 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef201405e0 con 0x7fef20073d40 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2055731316 0 0) 0x7fef201408c0 con 0x7fef200a5810 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 >> v1:172.21.15.45:6790/0 conn(0x7fef200a4d20 legacy=0x7fef201d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 >> v1:172.21.15.46:6789/0 conn(0x7fef20073d40 legacy=0x7fef201cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient: _finish_hunting 0 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 monclient: found mon.a 2024-09-16T03:23:56.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef200a1510 con 0x7fef200a5810 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient: _finish_auth 0 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient: _check_auth_tickets 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 5 monclient: authenticate success, global_id 174291 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 monclient: _renew_subs 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef20074d90 con 0x7fef200a5810 2024-09-16T03:23:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 client.174291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 monclient: _renew_subs 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fef2013dab0 con 0x7fef200a5810 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 1 librados: init done 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef18003620 con 0x7fef200a5810 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fef18003ab0 con 0x7fef200a5810 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef248e9640 4 set_mon_vals no callback set 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 monclient: dump: 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fef18003c50 con 0x7fef200a5810 2024-09-16T03:23:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef00005090 con 0x7fef200a5810 2024-09-16T03:23:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7fef1804b510 con 0x7fef200a5810 2024-09-16T03:23:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 client.174291.objecter ms_dispatch 0x7fef20000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 3 client.174291.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 3 client.174291.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 20 client.174291.objecter dump_active .. 0 homeless 2024-09-16T03:23:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef260ec640 10 monclient: get_auth_request con 0x7fef0c03e330 auth_method 0 2024-09-16T03:23:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.019+0000 7fef1effd640 10 client.174291.objecter ms_handle_connect 0x7fef0c03e330 2024-09-16T03:23:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.023+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fef18016ad0 con 0x7fef200a5810 2024-09-16T03:23:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.023+0000 7fef1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.023+0000 7fef1effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:56.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.195+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fef00000ba0 con 0x7fef0c03e330 2024-09-16T03:23:56.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef1effd640 1 -- 172.21.15.45:0/3139410553 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7fef00000ba0 con 0x7fef0c03e330 2024-09-16T03:23:56.198 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T03:23:56.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 10 librados: watch_flush enter 2024-09-16T03:23:56.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 10 librados: watch_flush exit 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 20 client.174291.objecter shutdown clearing up homeless session... 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 10 client.174291.objecter successfully canceled tick 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 >> v1:172.21.15.46:6812/149360280 conn(0x7fef0c03e330 legacy=0x7fef0c0407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 10 monclient: shutdown 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 >> v1:172.21.15.45:6789/0 conn(0x7fef200a5810 legacy=0x7fef201d5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 shutdown_connections 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 >> 172.21.15.45:0/3139410553 conn(0x7fef200aa720 msgr2=0x7fef200a4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 shutdown_connections 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 1 -- 172.21.15.45:0/3139410553 wait complete. 2024-09-16T03:23:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.199+0000 7fef28350640 1 librados: shutdown 2024-09-16T03:23:56.216 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T03:23:56.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:23:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient: get_monmap_and_config 2024-09-16T03:23:56.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient: build_initial_monmap 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient: monmap: 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:56.348765+0000 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:56.348765+0000 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:56.347 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 Processor -- start 2024-09-16T03:23:56.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 -- start start 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient: init 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient(hunting): picked mon.noname-b con 0x7fbef0135ab0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient(hunting): picked mon.noname-a con 0x7fbef009dee0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient(hunting): picked mon.noname-c con 0x7fbef013c030 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef009eee0 con 0x7fbef009dee0 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef009f1c0 con 0x7fbef013c030 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef009f4a0 con 0x7fbef0135ab0 2024-09-16T03:23:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient(hunting): authenticate will time out at 2126.998149s 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fbef0135ab0 0x7fbef0135e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58576/0 (socket says 172.21.15.45:58576) 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeffff640 1 -- 172.21.15.45:0/2006243324 learned_addr learned my addr 172.21.15.45:0/2006243324 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076157317 0 0) 0x7fbef009f4a0 con 0x7fbef0135ab0 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient(hunting): my global_id is 184187 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbed8003530 con 0x7fbef0135ab0 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3463353058 0 0) 0x7fbef009eee0 con 0x7fbef009dee0 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient(hunting): my global_id is 174294 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbef009f4a0 con 0x7fbef009dee0 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 582199984 0 0) 0x7fbed8003530 con 0x7fbef0135ab0 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 >> v1:172.21.15.46:6789/0 conn(0x7fbef013c030 legacy=0x7fbef013e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 >> v1:172.21.15.45:6789/0 conn(0x7fbef009dee0 legacy=0x7fbef013a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: _finish_hunting 0 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 monclient: found mon.noname-b 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef009dc30 con 0x7fbef0135ab0 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: _finish_auth 0 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:23:56.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbee00016a0 con 0x7fbef0135ab0 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: dump: 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.361 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 5 monclient: authenticate success, global_id 184187 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbee0001840 con 0x7fbef0135ab0 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 1 -- 172.21.15.45:0/2006243324 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbee0001c10 con 0x7fbef0135ab0 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbeeeffd640 10 monclient: dump: 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.362 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 4 set_mon_vals no callback set 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 10 monclient: shutdown 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 -- 172.21.15.45:0/2006243324 >> v1:172.21.15.45:6790/0 conn(0x7fbef0135ab0 legacy=0x7fbef0135e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 -- 172.21.15.45:0/2006243324 shutdown_connections 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 -- 172.21.15.45:0/2006243324 >> 172.21.15.45:0/2006243324 conn(0x7fbef00a65a0 msgr2=0x7fbef00a29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 -- 172.21.15.45:0/2006243324 shutdown_connections 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.347+0000 7fbef661c640 1 -- 172.21.15.45:0/2006243324 wait complete. 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: build_initial_monmap 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: monmap: 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 librados: starting msgr at 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 librados: starting objecter 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 Processor -- start 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 -- start start 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 librados: setting wanted keys 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 librados: calling monclient init 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: init 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient(hunting): picked mon.noname-a con 0x7fbef009dee0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient(hunting): picked mon.noname-c con 0x7fbef0135ab0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient(hunting): picked mon.noname-b con 0x7fbef013c030 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:56.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef0073b60 con 0x7fbef009dee0 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef006f750 con 0x7fbef013c030 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef0073640 con 0x7fbef0135ab0 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient(hunting): authenticate will time out at 2127.001281s 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeeffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbef009dee0 0x7fbef0072800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60410/0 (socket says 172.21.15.45:60410) 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeeffff640 1 -- 172.21.15.45:0/3648623056 learned_addr learned my addr 172.21.15.45:0/3648623056 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 client.?.objecter ms_handle_connect 0x7fbef009dee0 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 client.?.objecter ms_handle_connect 0x7fbef0135ab0 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100762231 0 0) 0x7fbef0073640 con 0x7fbef0135ab0 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): my global_id is 184190 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbedc004ec0 con 0x7fbef0135ab0 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744517889 0 0) 0x7fbef0073b60 con 0x7fbef009dee0 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): my global_id is 174297 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbef0073640 con 0x7fbef009dee0 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 client.?.objecter ms_handle_connect 0x7fbef013c030 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1405101690 0 0) 0x7fbef0073640 con 0x7fbef009dee0 2024-09-16T03:23:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbef0073b60 con 0x7fbef009dee0 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 599012723 0 0) 0x7fbedc004ec0 con 0x7fbef0135ab0 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbef0073640 con 0x7fbef0135ab0 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbee0001560 con 0x7fbef009dee0 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): dump: 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.368 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbee4001860 con 0x7fbef0135ab0 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): dump: 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.369 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015768805 0 0) 0x7fbef006f750 con 0x7fbef013c030 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient(hunting): my global_id is 164377 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbedc004ec0 con 0x7fbef013c030 2024-09-16T03:23:56.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1717215384 0 0) 0x7fbef0073b60 con 0x7fbef009dee0 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 >> v1:172.21.15.45:6790/0 conn(0x7fbef0135ab0 legacy=0x7fbef0072f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 >> v1:172.21.15.46:6789/0 conn(0x7fbef013c030 legacy=0x7fbef01d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient: _finish_hunting 0 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 monclient: found mon.a 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef013bb60 con 0x7fbef009dee0 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient: _finish_auth 0 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient: _check_auth_tickets 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 5 monclient: authenticate success, global_id 174297 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: _renew_subs 2024-09-16T03:23:56.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbef006cf80 con 0x7fbef009dee0 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 client.174297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: _renew_subs 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbef0070060 con 0x7fbef009dee0 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 librados: init done 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbee0001800 con 0x7fbef009dee0 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbee0004c40 con 0x7fbef009dee0 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient: dump: 2024-09-16T03:23:56.372 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeee7fc640 4 set_mon_vals no callback set 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 client.174297.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 15 client.174297.objecter _session_command_op_assign -1 1 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 20 client.174297.objecter _get_session osd=-1 returning homeless 2024-09-16T03:23:56.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 client.174297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.45:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbed0002920 con 0x7fbef009dee0 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fbee0010cb0 con 0x7fbef009dee0 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7fbee004b2f0 con 0x7fbef009dee0 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 client.174297.objecter ms_dispatch 0x7fbef0000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 3 client.174297.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 3 client.174297.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 client.174297.objecter checking command 1 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 20 client.174297.objecter _get_session s=0x7fbedc0461f0 osd=3 3 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 20 client.174297.objecter put_session s=0x7fbedc0461f0 osd=3 3 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 15 client.174297.objecter _session_command_op_remove -1 1 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 20 client.174297.objecter _get_session s=0x7fbedc0461f0 osd=3 3 2024-09-16T03:23:56.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 20 client.174297.objecter get_session s=0x7fbedc0461f0 osd=3 3 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 15 client.174297.objecter _session_command_op_assign 3 1 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 20 client.174297.objecter put_session s=0x7fbedc0461f0 osd=3 4 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 client.174297.objecter _send_command 1 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.46:6808/3088806475 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbedc04a5b0 con 0x7fbedc0468a0 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 20 client.174297.objecter dump_active .. 0 homeless 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 1851051592 0 0) 0x7fbee004b740 con 0x7fbef009dee0 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeecff9640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbeef7fe640 10 monclient: get_auth_request con 0x7fbedc03e2f0 auth_method 0 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.351+0000 7fbef4bb9640 10 monclient: get_auth_request con 0x7fbedc0468a0 auth_method 0 2024-09-16T03:23:56.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.355+0000 7fbeecff9640 10 client.174297.objecter ms_handle_connect 0x7fbedc03e2f0 2024-09-16T03:23:56.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.355+0000 7fbeecff9640 10 client.174297.objecter ms_handle_connect 0x7fbedc0468a0 2024-09-16T03:23:56.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.355+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== osd.3 v1:172.21.15.46:6808/3088806475 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbedc04a5b0 con 0x7fbedc0468a0 2024-09-16T03:23:56.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.355+0000 7fbeecff9640 10 client.174297.objecter ms_dispatch 0x7fbef0000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:23:56.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.355+0000 7fbeecff9640 10 client.174297.objecter _finish_command 1 = system:0 2024-09-16T03:23:56.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.355+0000 7fbeecff9640 20 client.174297.objecter put_session s=0x7fbedc0461f0 osd=3 4 2024-09-16T03:23:56.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.355+0000 7fbeecff9640 15 client.174297.objecter _session_command_op_remove 3 1 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 10 client.174297.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 15 client.174297.objecter _session_command_op_assign -1 2 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 20 client.174297.objecter _get_session s=0x7fbedc0461f0 osd=3 3 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 20 client.174297.objecter put_session s=0x7fbedc0461f0 osd=3 3 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 20 client.174297.objecter _get_session s=0x7fbedc0461f0 osd=3 3 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 15 client.174297.objecter _session_command_op_remove -1 2 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 20 client.174297.objecter get_session s=0x7fbedc0461f0 osd=3 3 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 15 client.174297.objecter _session_command_op_assign 3 2 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 20 client.174297.objecter put_session s=0x7fbedc0461f0 osd=3 4 2024-09-16T03:23:56.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 10 client.174297.objecter _send_command 2 2024-09-16T03:23:56.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.367+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 --> v1:172.21.15.46:6808/3088806475 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbed00012c0 con 0x7fbedc0468a0 2024-09-16T03:23:56.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbeecff9640 1 -- 172.21.15.45:0/3648623056 <== osd.3 v1:172.21.15.46:6808/3088806475 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fbed00012c0 con 0x7fbedc0468a0 2024-09-16T03:23:56.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbeecff9640 10 client.174297.objecter ms_dispatch 0x7fbef0000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:23:56.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbeecff9640 10 client.174297.objecter _finish_command 2 = system:0 2024-09-16T03:23:56.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbeecff9640 20 client.174297.objecter put_session s=0x7fbedc0461f0 osd=3 4 2024-09-16T03:23:56.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbeecff9640 15 client.174297.objecter _session_command_op_remove 3 2 2024-09-16T03:23:56.378 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:23:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:23:56.379 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:23:56.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 10 librados: watch_flush enter 2024-09-16T03:23:56.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 10 librados: watch_flush exit 2024-09-16T03:23:56.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 10 client.174297.objecter close_session for osd.3 2024-09-16T03:23:56.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 >> v1:172.21.15.46:6808/3088806475 conn(0x7fbedc0468a0 legacy=0x7fbedc048d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 20 client.174297.objecter put_session s=0x7fbedc0461f0 osd=3 1 2024-09-16T03:23:56.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 20 client.174297.objecter shutdown clearing up homeless session... 2024-09-16T03:23:56.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 10 client.174297.objecter successfully canceled tick 2024-09-16T03:23:56.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 >> v1:172.21.15.46:6812/149360280 conn(0x7fbedc03e2f0 legacy=0x7fbedc0407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 10 monclient: shutdown 2024-09-16T03:23:56.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:56.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 >> v1:172.21.15.45:6789/0 conn(0x7fbef009dee0 legacy=0x7fbef0072800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef4bb9640 1 -- 172.21.15.45:0/3648623056 reap_dead start 2024-09-16T03:23:56.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 shutdown_connections 2024-09-16T03:23:56.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 >> 172.21.15.45:0/3648623056 conn(0x7fbef00a65a0 msgr2=0x7fbef012e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:56.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 shutdown_connections 2024-09-16T03:23:56.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 1 -- 172.21.15.45:0/3648623056 wait complete. 2024-09-16T03:23:56.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.371+0000 7fbef661c640 1 librados: shutdown 2024-09-16T03:23:56.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T03:23:56.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.499+0000 7f44877ff640 10 monclient: get_monmap_and_config 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.499+0000 7f44877ff640 10 monclient: build_initial_monmap 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.499+0000 7f44877ff640 10 monclient: monmap: 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:56.503876+0000 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:56.503876+0000 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:56.502 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:56.503 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 Processor -- start 2024-09-16T03:23:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 -- start start 2024-09-16T03:23:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient: init 2024-09-16T03:23:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient(hunting): picked mon.noname-b con 0x7f4480135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient(hunting): picked mon.noname-a con 0x7f448009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient(hunting): picked mon.noname-c con 0x7f448013c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448009f7e0 con 0x7f448009fc40 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448009b160 con 0x7f448013c010 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448009b440 con 0x7f4480135950 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient(hunting): authenticate will time out at 2127.153261s 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f448559b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4480135950 0x7f4480135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58606/0 (socket says 172.21.15.45:58606) 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f448559b640 1 -- 172.21.15.45:0/2248400555 learned_addr learned my addr 172.21.15.45:0/2248400555 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4484d9a640 1 --1- 172.21.15.45:0/2248400555 >> v1:172.21.15.45:6789/0 conn(0x7f448009fc40 0x7f448013a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60426/0 (socket says 172.21.15.45:60426) 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395186194 0 0) 0x7f448009b440 con 0x7f4480135950 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient(hunting): my global_id is 184193 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f446c003530 con 0x7f4480135950 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3540247530 0 0) 0x7f448009f7e0 con 0x7f448009fc40 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient(hunting): my global_id is 174300 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f448009b440 con 0x7f448009fc40 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 474691819 0 0) 0x7f446c003530 con 0x7f4480135950 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 >> v1:172.21.15.46:6789/0 conn(0x7f448013c010 legacy=0x7f448013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 >> v1:172.21.15.45:6789/0 conn(0x7f448009fc40 legacy=0x7f448013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: _finish_hunting 0 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 monclient: found mon.noname-b 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4480136440 con 0x7f4480135950 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: _finish_auth 0 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: _check_auth_tickets 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f447c0015d0 con 0x7f4480135950 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: dump: 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 5 monclient: authenticate success, global_id 184193 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f447c001770 con 0x7f4480135950 2024-09-16T03:23:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 4 set_mon_vals no callback set 2024-09-16T03:23:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 1 -- 172.21.15.45:0/2248400555 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f447c001c00 con 0x7f4480135950 2024-09-16T03:23:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f4477fff640 10 monclient: dump: 2024-09-16T03:23:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient: shutdown 2024-09-16T03:23:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 -- 172.21.15.45:0/2248400555 >> v1:172.21.15.45:6790/0 conn(0x7f4480135950 legacy=0x7f4480135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 -- 172.21.15.45:0/2248400555 shutdown_connections 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 -- 172.21.15.45:0/2248400555 >> 172.21.15.45:0/2248400555 conn(0x7f44800a2bd0 msgr2=0x7f44800a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 -- 172.21.15.45:0/2248400555 shutdown_connections 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 -- 172.21.15.45:0/2248400555 wait complete. 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient: build_initial_monmap 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 10 monclient: monmap: 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.518 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:56.518 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:56.518 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:56.518 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 librados: starting msgr at 2024-09-16T03:23:56.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 librados: starting objecter 2024-09-16T03:23:56.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.503+0000 7f44877ff640 1 Processor -- start 2024-09-16T03:23:56.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 -- start start 2024-09-16T03:23:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 librados: setting wanted keys 2024-09-16T03:23:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 librados: calling monclient init 2024-09-16T03:23:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient: init 2024-09-16T03:23:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:23:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient(hunting): picked mon.noname-a con 0x7f448009fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient(hunting): picked mon.noname-b con 0x7f4480135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient(hunting): picked mon.noname-c con 0x7f448013c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44800712c0 con 0x7f448009fc40 2024-09-16T03:23:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44800715a0 con 0x7f4480135950 2024-09-16T03:23:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448013b590 con 0x7f448013c010 2024-09-16T03:23:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f448559b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f448009fc40 0x7f448006e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60440/0 (socket says 172.21.15.45:60440) 2024-09-16T03:23:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f448559b640 1 -- 172.21.15.45:0/2589418645 learned_addr learned my addr 172.21.15.45:0/2589418645 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient(hunting): authenticate will time out at 2127.156695s 2024-09-16T03:23:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 client.?.objecter ms_handle_connect 0x7f448009fc40 2024-09-16T03:23:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 client.?.objecter ms_handle_connect 0x7f448013c010 2024-09-16T03:23:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550341913 0 0) 0x7f44800712c0 con 0x7f448009fc40 2024-09-16T03:23:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): my global_id is 174303 2024-09-16T03:23:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4464004ec0 con 0x7f448009fc40 2024-09-16T03:23:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3801461874 0 0) 0x7f448013b590 con 0x7f448013c010 2024-09-16T03:23:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): my global_id is 184196 2024-09-16T03:23:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44800712c0 con 0x7f448013c010 2024-09-16T03:23:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 client.?.objecter ms_handle_connect 0x7f4480135950 2024-09-16T03:23:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 668074540 0 0) 0x7f4464004ec0 con 0x7f448009fc40 2024-09-16T03:23:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f448013b590 con 0x7f448009fc40 2024-09-16T03:23:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f447c001bc0 con 0x7f448009fc40 2024-09-16T03:23:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): dump: 2024-09-16T03:23:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2596968739 0 0) 0x7f44800712c0 con 0x7f448013c010 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4464004ec0 con 0x7f448013c010 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4478001bc0 con 0x7f448013c010 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): dump: 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3721226735 0 0) 0x7f44800715a0 con 0x7f4480135950 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient(hunting): my global_id is 164383 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44800712c0 con 0x7f4480135950 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 703601989 0 0) 0x7f448013b590 con 0x7f448009fc40 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 >> v1:172.21.15.45:6790/0 conn(0x7f448013c010 legacy=0x7f44801d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 >> v1:172.21.15.46:6789/0 conn(0x7f4480135950 legacy=0x7f448006ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 monclient: found mon.a 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4480070360 con 0x7f448009fc40 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient: _finish_auth 0 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:23:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 5 monclient: authenticate success, global_id 174303 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient: _renew_subs 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f448006f430 con 0x7f448009fc40 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 client.174303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient: _renew_subs 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f44801d4470 con 0x7f448009fc40 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 librados: init done 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f447c001710 con 0x7f448009fc40 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f447c004c40 con 0x7f448009fc40 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44777fe640 4 set_mon_vals no callback set 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 monclient: dump: 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:56.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4460005090 con 0x7f448009fc40 2024-09-16T03:23:56.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f447c004de0 con 0x7f448009fc40 2024-09-16T03:23:56.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f447c04b490 con 0x7f448009fc40 2024-09-16T03:23:56.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 client.174303.objecter ms_dispatch 0x7f4480000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:56.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 3 client.174303.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:56.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 3 client.174303.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:56.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 20 client.174303.objecter dump_active .. 0 homeless 2024-09-16T03:23:56.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4484d9a640 10 monclient: get_auth_request con 0x7f446403e2c0 auth_method 0 2024-09-16T03:23:56.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.507+0000 7f4475ffb640 10 client.174303.objecter ms_handle_connect 0x7f446403e2c0 2024-09-16T03:23:56.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.511+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f447c016b30 con 0x7f448009fc40 2024-09-16T03:23:56.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.511+0000 7f4475ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:56.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.511+0000 7f4475ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:56.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f4460000ba0 con 0x7f446403e2c0 2024-09-16T03:23:56.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f4475ffb640 1 -- 172.21.15.45:0/2589418645 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7f4460000ba0 con 0x7f446403e2c0 2024-09-16T03:23:56.686 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T03:23:56.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 10 librados: watch_flush enter 2024-09-16T03:23:56.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 10 librados: watch_flush exit 2024-09-16T03:23:56.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 20 client.174303.objecter shutdown clearing up homeless session... 2024-09-16T03:23:56.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 10 client.174303.objecter successfully canceled tick 2024-09-16T03:23:56.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 >> v1:172.21.15.46:6812/149360280 conn(0x7f446403e2c0 legacy=0x7f4464040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 10 monclient: shutdown 2024-09-16T03:23:56.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:56.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 >> v1:172.21.15.45:6789/0 conn(0x7f448009fc40 legacy=0x7f448006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 shutdown_connections 2024-09-16T03:23:56.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 >> 172.21.15.45:0/2589418645 conn(0x7f44800a2bd0 msgr2=0x7f448013ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:56.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 shutdown_connections 2024-09-16T03:23:56.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 1 -- 172.21.15.45:0/2589418645 wait complete. 2024-09-16T03:23:56.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.687+0000 7f44877ff640 1 librados: shutdown 2024-09-16T03:23:56.705 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T03:23:56.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:23:56.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.835+0000 7f4b95407640 10 monclient: get_monmap_and_config 2024-09-16T03:23:56.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.835+0000 7f4b95407640 10 monclient: build_initial_monmap 2024-09-16T03:23:56.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.835+0000 7f4b95407640 10 monclient: monmap: 2024-09-16T03:23:56.837 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:56.837 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.837 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:56.839544+0000 2024-09-16T03:23:56.837 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:56.839544+0000 2024-09-16T03:23:56.838 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:56.838 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.838 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:56.838 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:56.838 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:56.838 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 Processor -- start 2024-09-16T03:23:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 -- start start 2024-09-16T03:23:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient: init 2024-09-16T03:23:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient(hunting): picked mon.noname-a con 0x7f4b900a74a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient(hunting): picked mon.noname-b con 0x7f4b900a4580 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient(hunting): picked mon.noname-c con 0x7f4b90073c30 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b90137fd0 con 0x7f4b900a74a0 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b90074cd0 con 0x7f4b90073c30 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b90073980 con 0x7f4b900a4580 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient(hunting): authenticate will time out at 2127.489114s 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4b900a74a0 0x7f4b900a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60442/0 (socket says 172.21.15.45:60442) 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8effd640 1 -- 172.21.15.45:0/1732240553 learned_addr learned my addr 172.21.15.45:0/1732240553 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3099264359 0 0) 0x7f4b90137fd0 con 0x7f4b900a74a0 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): my global_id is 174306 2024-09-16T03:23:56.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b78003530 con 0x7f4b900a74a0 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3888100014 0 0) 0x7f4b90073980 con 0x7f4b900a4580 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): my global_id is 184199 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b90137fd0 con 0x7f4b900a4580 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963310300 0 0) 0x7f4b90074cd0 con 0x7f4b90073c30 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient(hunting): my global_id is 164386 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b90073980 con 0x7f4b90073c30 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4178977976 0 0) 0x7f4b78003530 con 0x7f4b900a74a0 2024-09-16T03:23:56.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 >> v1:172.21.15.45:6790/0 conn(0x7f4b900a4580 legacy=0x7f4b9013cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 >> v1:172.21.15.46:6789/0 conn(0x7f4b90073c30 legacy=0x7f4b90140510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 monclient: found mon.noname-a 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b900a0cf0 con 0x7f4b900a74a0 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: _finish_auth 0 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4b80001a20 con 0x7f4b900a74a0 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: dump: 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 5 monclient: authenticate success, global_id 174306 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b80001c40 con 0x7f4b900a74a0 2024-09-16T03:23:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 4 set_mon_vals no callback set 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 1 -- 172.21.15.45:0/1732240553 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4b80002090 con 0x7f4b900a74a0 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b8dffb640 10 monclient: dump: 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 10 monclient: shutdown 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 -- 172.21.15.45:0/1732240553 >> v1:172.21.15.45:6789/0 conn(0x7f4b900a74a0 legacy=0x7f4b900a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 -- 172.21.15.45:0/1732240553 shutdown_connections 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 -- 172.21.15.45:0/1732240553 >> 172.21.15.45:0/1732240553 conn(0x7f4b900aa890 msgr2=0x7f4b900a6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 -- 172.21.15.45:0/1732240553 shutdown_connections 2024-09-16T03:23:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.839+0000 7f4b95407640 1 -- 172.21.15.45:0/1732240553 wait complete. 2024-09-16T03:23:56.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: build_initial_monmap 2024-09-16T03:23:56.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: monmap: 2024-09-16T03:23:56.852 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:56.852 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:56.852 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 librados: starting msgr at 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 librados: starting objecter 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 Processor -- start 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 -- start start 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 librados: setting wanted keys 2024-09-16T03:23:56.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 librados: calling monclient init 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: init 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient(hunting): picked mon.noname-a con 0x7f4b90073c30 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient(hunting): picked mon.noname-c con 0x7f4b900a4580 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient(hunting): picked mon.noname-b con 0x7f4b901d32a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b90134fd0 con 0x7f4b90073c30 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b90132de0 con 0x7f4b901d32a0 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b901352b0 con 0x7f4b900a4580 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b8effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4b90073c30 0x7f4b90131fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60456/0 (socket says 172.21.15.45:60456) 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient(hunting): authenticate will time out at 2127.493514s 2024-09-16T03:23:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b8effd640 1 -- 172.21.15.45:0/4084624635 learned_addr learned my addr 172.21.15.45:0/4084624635 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 client.?.objecter ms_handle_connect 0x7f4b90073c30 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 client.?.objecter ms_handle_connect 0x7f4b900a4580 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 744871126 0 0) 0x7f4b90134fd0 con 0x7f4b90073c30 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): my global_id is 174309 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b7c004ec0 con 0x7f4b90073c30 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3016871804 0 0) 0x7f4b901352b0 con 0x7f4b900a4580 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): my global_id is 184202 2024-09-16T03:23:56.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b90134fd0 con 0x7f4b900a4580 2024-09-16T03:23:56.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 client.?.objecter ms_handle_connect 0x7f4b901d32a0 2024-09-16T03:23:56.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:56.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3313587869 0 0) 0x7f4b7c004ec0 con 0x7f4b90073c30 2024-09-16T03:23:56.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b901352b0 con 0x7f4b90073c30 2024-09-16T03:23:56.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4b80001f90 con 0x7f4b90073c30 2024-09-16T03:23:56.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): dump: 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2455905603 0 0) 0x7f4b90132de0 con 0x7f4b901d32a0 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): my global_id is 164389 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b7c004ec0 con 0x7f4b901d32a0 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1434074960 0 0) 0x7f4b90134fd0 con 0x7f4b900a4580 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b90132de0 con 0x7f4b900a4580 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4b840018c0 con 0x7f4b900a4580 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient(hunting): dump: 2024-09-16T03:23:56.858 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 214907640 0 0) 0x7f4b901352b0 con 0x7f4b90073c30 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 >> v1:172.21.15.45:6790/0 conn(0x7f4b900a4580 legacy=0x7f4b901326d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 >> v1:172.21.15.46:6789/0 conn(0x7f4b901d32a0 legacy=0x7f4b901d5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient: _finish_hunting 0 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 monclient: found mon.a 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b900a9d70 con 0x7f4b90073c30 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient: _finish_auth 0 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient: _check_auth_tickets 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 5 monclient: authenticate success, global_id 174309 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: _renew_subs 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b90134cf0 con 0x7f4b90073c30 2024-09-16T03:23:56.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 client.174309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: _renew_subs 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b900a0400 con 0x7f4b90073c30 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 librados: init done 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b80001550 con 0x7f4b90073c30 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4b80005000 con 0x7f4b90073c30 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b8d7fa640 4 set_mon_vals no callback set 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 10 monclient: dump: 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:56.861 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 client.174309.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 15 client.174309.objecter _session_command_op_assign -1 1 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 20 client.174309.objecter _get_session osd=-1 returning homeless 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 client.174309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:56.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.45:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4b70002920 con 0x7f4b90073c30 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.843+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4b80011070 con 0x7f4b90073c30 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f4b8004b820 con 0x7f4b90073c30 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 10 client.174309.objecter ms_dispatch 0x7f4b90000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 3 client.174309.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 3 client.174309.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 10 client.174309.objecter checking command 1 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 20 client.174309.objecter _get_session s=0x7f4b7c0461a0 osd=4 3 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 20 client.174309.objecter put_session s=0x7f4b7c0461a0 osd=4 3 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 15 client.174309.objecter _session_command_op_remove -1 1 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 20 client.174309.objecter _get_session s=0x7f4b7c0461a0 osd=4 3 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 20 client.174309.objecter get_session s=0x7f4b7c0461a0 osd=4 3 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 15 client.174309.objecter _session_command_op_assign 4 1 2024-09-16T03:23:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 20 client.174309.objecter put_session s=0x7f4b7c0461a0 osd=4 4 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 10 client.174309.objecter _send_command 1 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.46:6800/264199318 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b7c04a560 con 0x7f4b7c046850 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 20 client.174309.objecter dump_active .. 0 homeless 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 1851051592 0 0) 0x7f4b8004bc30 con 0x7f4b90073c30 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b8e7fc640 10 monclient: get_auth_request con 0x7f4b7c03e2b0 auth_method 0 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b8f7fe640 10 monclient: get_auth_request con 0x7f4b7c046850 auth_method 0 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 10 client.174309.objecter ms_handle_connect 0x7f4b7c03e2b0 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.847+0000 7f4b777fe640 10 client.174309.objecter ms_handle_connect 0x7f4b7c046850 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.851+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== osd.4 v1:172.21.15.46:6800/264199318 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b7c04a560 con 0x7f4b7c046850 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.851+0000 7f4b777fe640 10 client.174309.objecter ms_dispatch 0x7f4b90000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.851+0000 7f4b777fe640 10 client.174309.objecter _finish_command 1 = system:0 2024-09-16T03:23:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.851+0000 7f4b777fe640 20 client.174309.objecter put_session s=0x7f4b7c0461a0 osd=4 4 2024-09-16T03:23:56.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.851+0000 7f4b777fe640 15 client.174309.objecter _session_command_op_remove 4 1 2024-09-16T03:23:56.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 10 client.174309.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T03:23:56.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 15 client.174309.objecter _session_command_op_assign -1 2 2024-09-16T03:23:56.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 20 client.174309.objecter _get_session s=0x7f4b7c0461a0 osd=4 3 2024-09-16T03:23:56.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 20 client.174309.objecter put_session s=0x7f4b7c0461a0 osd=4 3 2024-09-16T03:23:56.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 20 client.174309.objecter _get_session s=0x7f4b7c0461a0 osd=4 3 2024-09-16T03:23:56.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 15 client.174309.objecter _session_command_op_remove -1 2 2024-09-16T03:23:56.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 20 client.174309.objecter get_session s=0x7f4b7c0461a0 osd=4 3 2024-09-16T03:23:56.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 15 client.174309.objecter _session_command_op_assign 4 2 2024-09-16T03:23:56.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 20 client.174309.objecter put_session s=0x7f4b7c0461a0 osd=4 4 2024-09-16T03:23:56.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 10 client.174309.objecter _send_command 2 2024-09-16T03:23:56.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.863+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 --> v1:172.21.15.46:6800/264199318 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4b700012c0 con 0x7f4b7c046850 2024-09-16T03:23:56.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b777fe640 1 -- 172.21.15.45:0/4084624635 <== osd.4 v1:172.21.15.46:6800/264199318 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f4b700012c0 con 0x7f4b7c046850 2024-09-16T03:23:56.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b777fe640 10 client.174309.objecter ms_dispatch 0x7f4b90000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:23:56.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b777fe640 10 client.174309.objecter _finish_command 2 = system:0 2024-09-16T03:23:56.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b777fe640 20 client.174309.objecter put_session s=0x7f4b7c0461a0 osd=4 4 2024-09-16T03:23:56.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b777fe640 15 client.174309.objecter _session_command_op_remove 4 2 2024-09-16T03:23:56.867 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:23:56.867 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:23:56.867 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:23:56.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 10 librados: watch_flush enter 2024-09-16T03:23:56.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 10 librados: watch_flush exit 2024-09-16T03:23:56.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 10 client.174309.objecter close_session for osd.4 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 >> v1:172.21.15.46:6800/264199318 conn(0x7f4b7c046850 legacy=0x7f4b7c048cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 20 client.174309.objecter put_session s=0x7f4b7c0461a0 osd=4 1 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 20 client.174309.objecter shutdown clearing up homeless session... 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 10 client.174309.objecter successfully canceled tick 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 >> v1:172.21.15.46:6812/149360280 conn(0x7f4b7c03e2b0 legacy=0x7f4b7c040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 10 monclient: shutdown 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 >> v1:172.21.15.45:6789/0 conn(0x7f4b90073c30 legacy=0x7f4b90131fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b8f7fe640 1 -- 172.21.15.45:0/4084624635 reap_dead start 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 shutdown_connections 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 >> 172.21.15.45:0/4084624635 conn(0x7f4b900aa890 msgr2=0x7f4b900a8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 shutdown_connections 2024-09-16T03:23:56.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 1 -- 172.21.15.45:0/4084624635 wait complete. 2024-09-16T03:23:56.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:56.867+0000 7f4b95407640 1 librados: shutdown 2024-09-16T03:23:56.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T03:23:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.003+0000 7fd46a325640 10 monclient: get_monmap_and_config 2024-09-16T03:23:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.003+0000 7fd46a325640 10 monclient: build_initial_monmap 2024-09-16T03:23:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.003+0000 7fd46a325640 10 monclient: monmap: 2024-09-16T03:23:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:57.007429+0000 2024-09-16T03:23:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:57.007429+0000 2024-09-16T03:23:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:57.006 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.006 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:57.006 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:57.006 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:57.006 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 Processor -- start 2024-09-16T03:23:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- start start 2024-09-16T03:23:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient: init 2024-09-16T03:23:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient(hunting): picked mon.noname-b con 0x7fd464137d20 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient(hunting): picked mon.noname-a con 0x7fd464138100 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient(hunting): picked mon.noname-c con 0x7fd46413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4640a4f90 con 0x7fd464138100 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4640a4cb0 con 0x7fd46413e2a0 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd46409dc70 con 0x7fd464137d20 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient(hunting): authenticate will time out at 2127.656771s 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd463fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fd464137d20 0x7fd4640a5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58652/0 (socket says 172.21.15.45:58652) 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd463fff640 1 -- 172.21.15.45:0/1180002042 learned_addr learned my addr 172.21.15.45:0/1180002042 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 745446390 0 0) 0x7fd46409dc70 con 0x7fd464137d20 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient(hunting): my global_id is 184205 2024-09-16T03:23:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd44c003530 con 0x7fd464137d20 2024-09-16T03:23:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518718612 0 0) 0x7fd4640a4f90 con 0x7fd464138100 2024-09-16T03:23:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient(hunting): my global_id is 174312 2024-09-16T03:23:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd46409dc70 con 0x7fd464138100 2024-09-16T03:23:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1713421692 0 0) 0x7fd46409dc70 con 0x7fd464138100 2024-09-16T03:23:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 >> v1:172.21.15.45:6790/0 conn(0x7fd464137d20 legacy=0x7fd4640a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 >> v1:172.21.15.46:6789/0 conn(0x7fd46413e2a0 legacy=0x7fd464140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 monclient: found mon.noname-a 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46409cc50 con 0x7fd464138100 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: _finish_auth 0 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd458001680 con 0x7fd464138100 2024-09-16T03:23:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: dump: 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 5 monclient: authenticate success, global_id 174312 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd4580018a0 con 0x7fd464138100 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 1 -- 172.21.15.45:0/1180002042 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd458001d30 con 0x7fd464138100 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd462ffd640 10 monclient: dump: 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 4 set_mon_vals no callback set 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient: shutdown 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- 172.21.15.45:0/1180002042 >> v1:172.21.15.45:6789/0 conn(0x7fd464138100 legacy=0x7fd46413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- 172.21.15.45:0/1180002042 shutdown_connections 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- 172.21.15.45:0/1180002042 >> 172.21.15.45:0/1180002042 conn(0x7fd4640aa110 msgr2=0x7fd4640aa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- 172.21.15.45:0/1180002042 shutdown_connections 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- 172.21.15.45:0/1180002042 wait complete. 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient: build_initial_monmap 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient: monmap: 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 librados: starting msgr at 2024-09-16T03:23:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 librados: starting objecter 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 Processor -- start 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 -- start start 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 librados: setting wanted keys 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 1 librados: calling monclient init 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.007+0000 7fd46a325640 10 monclient: init 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient(hunting): picked mon.noname-a con 0x7fd464137d20 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient(hunting): picked mon.noname-c con 0x7fd464138100 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient(hunting): picked mon.noname-b con 0x7fd46413e2a0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4640a2120 con 0x7fd464137d20 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4640a3170 con 0x7fd46413e2a0 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4640a3f40 con 0x7fd464138100 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient(hunting): authenticate will time out at 2127.660201s 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd463fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd464137d20 0x7fd4640a1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60472/0 (socket says 172.21.15.45:60472) 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd463fff640 1 -- 172.21.15.45:0/3393354681 learned_addr learned my addr 172.21.15.45:0/3393354681 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd463fff640 1 -- 172.21.15.45:0/3393354681 >> v1:172.21.15.45:6789/0 conn(0x7fd464137d20 legacy=0x7fd4640a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd463fff640 1 -- 172.21.15.45:0/3393354681 >> v1:172.21.15.45:6789/0 conn(0x7fd464137d20 legacy=0x7fd4640a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd463fff640 1 --1- 172.21.15.45:0/3393354681 >> v1:172.21.15.45:6789/0 conn(0x7fd464137d20 0x7fd4640a1300 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd463fff640 1 --1- 172.21.15.45:0/3393354681 >> v1:172.21.15.45:6789/0 conn(0x7fd464137d20 0x7fd4640a1300 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.45:6789/0 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 client.?.objecter ms_handle_connect 0x7fd464138100 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200643126 0 0) 0x7fd4640a3f40 con 0x7fd464138100 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): my global_id is 184208 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd450004ec0 con 0x7fd464138100 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 client.?.objecter ms_handle_connect 0x7fd46413e2a0 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3144204843 0 0) 0x7fd450004ec0 con 0x7fd464138100 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4640a3f40 con 0x7fd464138100 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd458001bf0 con 0x7fd464138100 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): dump: 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.024 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200120133 0 0) 0x7fd4640a3170 con 0x7fd46413e2a0 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient(hunting): my global_id is 164395 2024-09-16T03:23:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd450004ec0 con 0x7fd46413e2a0 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1896628578 0 0) 0x7fd4640a3f40 con 0x7fd464138100 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 >> v1:172.21.15.46:6789/0 conn(0x7fd46413e2a0 legacy=0x7fd4641d4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 >> v1:172.21.15.45:6789/0 conn(0x7fd464137d20 legacy=0x7fd4640a1300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 monclient: found mon.c 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4640a4220 con 0x7fd464138100 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient: _finish_auth 0 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 5 monclient: authenticate success, global_id 184208 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient: _renew_subs 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd464137700 con 0x7fd464138100 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 client.184208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient: _renew_subs 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd464138f10 con 0x7fd464138100 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 1 librados: init done 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd458001840 con 0x7fd464138100 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fd458003b90 con 0x7fd464138100 2024-09-16T03:23:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:57.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd4627fc640 4 set_mon_vals no callback set 2024-09-16T03:23:57.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 monclient: dump: 2024-09-16T03:23:57.028 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.028 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.028 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.028 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fd458003d30 con 0x7fd464138100 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd444005090 con 0x7fd464138100 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7fd45804a430 con 0x7fd464138100 2024-09-16T03:23:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 client.184208.objecter ms_dispatch 0x7fd464000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 3 client.184208.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 3 client.184208.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 20 client.184208.objecter dump_active .. 0 homeless 2024-09-16T03:23:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd463fff640 10 monclient: get_auth_request con 0x7fd45003e010 auth_method 0 2024-09-16T03:23:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.011+0000 7fd460ff9640 10 client.184208.objecter ms_handle_connect 0x7fd45003e010 2024-09-16T03:23:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.015+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd458015a30 con 0x7fd464138100 2024-09-16T03:23:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.015+0000 7fd460ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.015+0000 7fd460ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:57.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.187+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fd444000ba0 con 0x7fd45003e010 2024-09-16T03:23:57.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.187+0000 7fd460ff9640 1 -- 172.21.15.45:0/3393354681 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7fd444000ba0 con 0x7fd45003e010 2024-09-16T03:23:57.189 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T03:23:57.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 10 librados: watch_flush enter 2024-09-16T03:23:57.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 10 librados: watch_flush exit 2024-09-16T03:23:57.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 20 client.184208.objecter shutdown clearing up homeless session... 2024-09-16T03:23:57.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 10 client.184208.objecter successfully canceled tick 2024-09-16T03:23:57.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 >> v1:172.21.15.46:6812/149360280 conn(0x7fd45003e010 legacy=0x7fd4500404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 10 monclient: shutdown 2024-09-16T03:23:57.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:57.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 >> v1:172.21.15.45:6790/0 conn(0x7fd464138100 legacy=0x7fd4640a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 shutdown_connections 2024-09-16T03:23:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 >> 172.21.15.45:0/3393354681 conn(0x7fd4640aa110 msgr2=0x7fd464130050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 shutdown_connections 2024-09-16T03:23:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 1 -- 172.21.15.45:0/3393354681 wait complete. 2024-09-16T03:23:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.191+0000 7fd46a325640 1 librados: shutdown 2024-09-16T03:23:57.209 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T03:23:57.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient: get_monmap_and_config 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient: build_initial_monmap 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient: monmap: 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:57.342099+0000 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:57.342099+0000 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:57.341 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 1 Processor -- start 2024-09-16T03:23:57.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 1 -- start start 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient: init 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd009f960 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd01326a0 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd013c1e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdd00a2a10 con 0x7fbdd01326a0 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdd00a00b0 con 0x7fbdd013c1e0 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdd006ef70 con 0x7fbdd009f960 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdd65f8640 10 monclient(hunting): authenticate will time out at 2127.991516s 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdcf7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbdd01326a0 0x7fbdd013aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60476/0 (socket says 172.21.15.45:60476) 2024-09-16T03:23:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.339+0000 7fbdcf7fe640 1 -- 172.21.15.45:0/1482511886 learned_addr learned my addr 172.21.15.45:0/1482511886 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2386951360 0 0) 0x7fbdd00a2a10 con 0x7fbdd01326a0 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): my global_id is 174315 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdbc003530 con 0x7fbdd01326a0 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3729060709 0 0) 0x7fbdd006ef70 con 0x7fbdd009f960 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): my global_id is 184211 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdd00a2a10 con 0x7fbdd009f960 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3238011910 0 0) 0x7fbdd00a00b0 con 0x7fbdd013c1e0 2024-09-16T03:23:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient(hunting): my global_id is 164398 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdd006ef70 con 0x7fbdd013c1e0 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 114689727 0 0) 0x7fbdbc003530 con 0x7fbdd01326a0 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 >> v1:172.21.15.45:6790/0 conn(0x7fbdd009f960 legacy=0x7fbdd00746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 >> v1:172.21.15.46:6789/0 conn(0x7fbdd013c1e0 legacy=0x7fbdd013e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: _finish_hunting 0 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 monclient: found mon.noname-a 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd006f250 con 0x7fbdd01326a0 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: _finish_auth 0 2024-09-16T03:23:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: _check_auth_tickets 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdb8001620 con 0x7fbdd01326a0 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: dump: 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.347 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 5 monclient: authenticate success, global_id 174315 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdb80017c0 con 0x7fbdd01326a0 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 1 -- 172.21.15.45:0/1482511886 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdb8001c50 con 0x7fbdd01326a0 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 4 set_mon_vals no callback set 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdceffd640 10 monclient: dump: 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.349 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: shutdown 2024-09-16T03:23:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1482511886 >> v1:172.21.15.45:6789/0 conn(0x7fbdd01326a0 legacy=0x7fbdd013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1482511886 shutdown_connections 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1482511886 >> 172.21.15.45:0/1482511886 conn(0x7fbdd00a6420 msgr2=0x7fbdd00a6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1482511886 shutdown_connections 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1482511886 wait complete. 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: build_initial_monmap 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: monmap: 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 librados: starting msgr at 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 librados: starting objecter 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 Processor -- start 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- start start 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 librados: setting wanted keys 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 librados: calling monclient init 2024-09-16T03:23:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: init 2024-09-16T03:23:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd009f960 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd01326a0 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd013c1e0 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdd0138dd0 con 0x7fbdd009f960 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdd009e530 con 0x7fbdd013c1e0 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdd013f910 con 0x7fbdd01326a0 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient(hunting): authenticate will time out at 2127.994747s 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdcf7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fbdd01326a0 0x7fbdd009de20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58674/0 (socket says 172.21.15.45:58674) 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdcf7fe640 1 -- 172.21.15.45:0/1075824369 learned_addr learned my addr 172.21.15.45:0/1075824369 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdcffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbdd009f960 0x7fbdd009d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60482/0 (socket says 172.21.15.45:60482) 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 client.?.objecter ms_handle_connect 0x7fbdd01326a0 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 client.?.objecter ms_handle_connect 0x7fbdd009f960 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1837767544 0 0) 0x7fbdd013f910 con 0x7fbdd01326a0 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): my global_id is 184214 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdb0004ec0 con 0x7fbdd01326a0 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3984708087 0 0) 0x7fbdd0138dd0 con 0x7fbdd009f960 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): my global_id is 174318 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdd013f910 con 0x7fbdd009f960 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 client.?.objecter ms_handle_connect 0x7fbdd013c1e0 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1965662525 0 0) 0x7fbdd013f910 con 0x7fbdd009f960 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdd0138dd0 con 0x7fbdd009f960 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2912715069 0 0) 0x7fbdb0004ec0 con 0x7fbdd01326a0 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdd013f910 con 0x7fbdd01326a0 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdc0001d80 con 0x7fbdd009f960 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): dump: 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdb80047f0 con 0x7fbdd01326a0 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): dump: 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228117387 0 0) 0x7fbdd009e530 con 0x7fbdd013c1e0 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient(hunting): my global_id is 164401 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdb0004ec0 con 0x7fbdd013c1e0 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 35734362 0 0) 0x7fbdd0138dd0 con 0x7fbdd009f960 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 >> v1:172.21.15.45:6790/0 conn(0x7fbdd01326a0 legacy=0x7fbdd009de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 >> v1:172.21.15.46:6789/0 conn(0x7fbdd013c1e0 legacy=0x7fbdd01d5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient: _finish_hunting 0 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 monclient: found mon.a 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd009d440 con 0x7fbdd009f960 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient: _finish_auth 0 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient: _check_auth_tickets 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 5 monclient: authenticate success, global_id 174318 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: _renew_subs 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd009f350 con 0x7fbdd009f960 2024-09-16T03:23:57.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 client.174318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:57.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: _renew_subs 2024-09-16T03:23:57.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:57.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbdd00740b0 con 0x7fbdd009f960 2024-09-16T03:23:57.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdd65f8640 1 librados: init done 2024-09-16T03:23:57.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdc0002060 con 0x7fbdd009f960 2024-09-16T03:23:57.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:57.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fbdc0003df0 con 0x7fbdd009f960 2024-09-16T03:23:57.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdce7fc640 4 set_mon_vals no callback set 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.343+0000 7fbdccff9640 10 monclient: dump: 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.363 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdd65f8640 10 client.174318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdd65f8640 15 client.174318.objecter _session_command_op_assign -1 1 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdd65f8640 20 client.174318.objecter _get_session osd=-1 returning homeless 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdd65f8640 10 client.174318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdd65f8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.45:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbdb4002920 con 0x7fbdd009f960 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fbdc0003f90 con 0x7fbdd009f960 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7fbdc004b840 con 0x7fbdd009f960 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 10 client.174318.objecter ms_dispatch 0x7fbdd0000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 3 client.174318.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 3 client.174318.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 10 client.174318.objecter checking command 1 2024-09-16T03:23:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 20 client.174318.objecter _get_session s=0x7fbdb0046110 osd=5 3 2024-09-16T03:23:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 20 client.174318.objecter put_session s=0x7fbdb0046110 osd=5 3 2024-09-16T03:23:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 15 client.174318.objecter _session_command_op_remove -1 1 2024-09-16T03:23:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 20 client.174318.objecter _get_session s=0x7fbdb0046110 osd=5 3 2024-09-16T03:23:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 20 client.174318.objecter get_session s=0x7fbdb0046110 osd=5 3 2024-09-16T03:23:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 15 client.174318.objecter _session_command_op_assign 5 1 2024-09-16T03:23:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 20 client.174318.objecter put_session s=0x7fbdb0046110 osd=5 4 2024-09-16T03:23:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 10 client.174318.objecter _send_command 1 2024-09-16T03:23:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.46:6804/3390152579 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdb004a4d0 con 0x7fbdb00467c0 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 20 client.174318.objecter dump_active .. 0 homeless 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 1851051592 0 0) 0x7fbdc004bc50 con 0x7fbdd009f960 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdcf7fe640 10 monclient: get_auth_request con 0x7fbdb003e270 auth_method 0 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdd4b95640 10 monclient: get_auth_request con 0x7fbdb00467c0 auth_method 0 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 10 client.174318.objecter ms_handle_connect 0x7fbdb003e270 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 10 client.174318.objecter ms_handle_connect 0x7fbdb00467c0 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== osd.5 v1:172.21.15.46:6804/3390152579 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbdb004a4d0 con 0x7fbdb00467c0 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 10 client.174318.objecter ms_dispatch 0x7fbdd0000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 10 client.174318.objecter _finish_command 1 = system:0 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 20 client.174318.objecter put_session s=0x7fbdb0046110 osd=5 4 2024-09-16T03:23:57.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.347+0000 7fbdccff9640 15 client.174318.objecter _session_command_op_remove 5 1 2024-09-16T03:23:57.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 10 client.174318.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T03:23:57.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 15 client.174318.objecter _session_command_op_assign -1 2 2024-09-16T03:23:57.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 20 client.174318.objecter _get_session s=0x7fbdb0046110 osd=5 3 2024-09-16T03:23:57.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 20 client.174318.objecter put_session s=0x7fbdb0046110 osd=5 3 2024-09-16T03:23:57.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 20 client.174318.objecter _get_session s=0x7fbdb0046110 osd=5 3 2024-09-16T03:23:57.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 15 client.174318.objecter _session_command_op_remove -1 2 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 20 client.174318.objecter get_session s=0x7fbdb0046110 osd=5 3 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 15 client.174318.objecter _session_command_op_assign 5 2 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 20 client.174318.objecter put_session s=0x7fbdb0046110 osd=5 4 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 10 client.174318.objecter _send_command 2 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 --> v1:172.21.15.46:6804/3390152579 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbdb40012c0 con 0x7fbdb00467c0 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdccff9640 1 -- 172.21.15.45:0/1075824369 <== osd.5 v1:172.21.15.46:6804/3390152579 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fbdb40012c0 con 0x7fbdb00467c0 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdccff9640 10 client.174318.objecter ms_dispatch 0x7fbdd0000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdccff9640 10 client.174318.objecter _finish_command 2 = system:0 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdccff9640 20 client.174318.objecter put_session s=0x7fbdb0046110 osd=5 4 2024-09-16T03:23:57.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdccff9640 15 client.174318.objecter _session_command_op_remove 5 2 2024-09-16T03:23:57.369 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-16T03:23:57.369 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:23:57.369 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-16T03:23:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 10 librados: watch_flush enter 2024-09-16T03:23:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 10 librados: watch_flush exit 2024-09-16T03:23:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 10 client.174318.objecter close_session for osd.5 2024-09-16T03:23:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 >> v1:172.21.15.46:6804/3390152579 conn(0x7fbdb00467c0 legacy=0x7fbdb0048c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 20 client.174318.objecter put_session s=0x7fbdb0046110 osd=5 1 2024-09-16T03:23:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 20 client.174318.objecter shutdown clearing up homeless session... 2024-09-16T03:23:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 10 client.174318.objecter successfully canceled tick 2024-09-16T03:23:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 >> v1:172.21.15.46:6812/149360280 conn(0x7fbdb003e270 legacy=0x7fbdb0040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 10 monclient: shutdown 2024-09-16T03:23:57.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:57.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 >> v1:172.21.15.45:6789/0 conn(0x7fbdd009f960 legacy=0x7fbdd009d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd4b95640 1 -- 172.21.15.45:0/1075824369 reap_dead start 2024-09-16T03:23:57.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 shutdown_connections 2024-09-16T03:23:57.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 >> 172.21.15.45:0/1075824369 conn(0x7fbdd00a6420 msgr2=0x7fbdd013eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:57.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 shutdown_connections 2024-09-16T03:23:57.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 1 -- 172.21.15.45:0/1075824369 wait complete. 2024-09-16T03:23:57.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.363+0000 7fbdd65f8640 1 librados: shutdown 2024-09-16T03:23:57.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T03:23:57.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.491+0000 7fb4b7fc6640 10 monclient: get_monmap_and_config 2024-09-16T03:23:57.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.491+0000 7fb4b7fc6640 10 monclient: build_initial_monmap 2024-09-16T03:23:57.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.491+0000 7fb4b7fc6640 10 monclient: monmap: 2024-09-16T03:23:57.493 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:57.493 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.494 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:57.495552+0000 2024-09-16T03:23:57.494 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:57.495552+0000 2024-09-16T03:23:57.494 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:57.494 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.494 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:57.494 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:57.494 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:57.494 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 Processor -- start 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- start start 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient: init 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b0135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b009fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b013c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b009f7e0 con 0x7fb4b013c010 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b009b160 con 0x7fb4b009fc40 2024-09-16T03:23:57.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b009b440 con 0x7fb4b0135950 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient(hunting): authenticate will time out at 2128.144955s 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b5d62640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb4b0135950 0x7fb4b0135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58680/0 (socket says 172.21.15.45:58680) 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b5d62640 1 -- 172.21.15.45:0/2602528688 learned_addr learned my addr 172.21.15.45:0/2602528688 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048628958 0 0) 0x7fb4b009f7e0 con 0x7fb4b013c010 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): my global_id is 174321 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb49c003530 con 0x7fb4b013c010 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 319719299 0 0) 0x7fb4b009b440 con 0x7fb4b0135950 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): my global_id is 184217 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4b009f7e0 con 0x7fb4b0135950 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503591000 0 0) 0x7fb4b009b160 con 0x7fb4b009fc40 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient(hunting): my global_id is 164404 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4b009b440 con 0x7fb4b009fc40 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 374190616 0 0) 0x7fb49c003530 con 0x7fb4b013c010 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 >> v1:172.21.15.45:6790/0 conn(0x7fb4b0135950 legacy=0x7fb4b0135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 >> v1:172.21.15.46:6789/0 conn(0x7fb4b009fc40 legacy=0x7fb4b013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: _finish_hunting 0 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 monclient: found mon.noname-a 2024-09-16T03:23:57.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b0136440 con 0x7fb4b013c010 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: _finish_auth 0 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: _check_auth_tickets 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb4a8001a20 con 0x7fb4b013c010 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 5 monclient: authenticate success, global_id 174321 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: dump: 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.504 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4a8001c40 con 0x7fb4b013c010 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 1 -- 172.21.15.45:0/2602528688 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb4a80020d0 con 0x7fb4b013c010 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b4d60640 10 monclient: dump: 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 4 set_mon_vals no callback set 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient: shutdown 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/2602528688 >> v1:172.21.15.45:6789/0 conn(0x7fb4b013c010 legacy=0x7fb4b013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/2602528688 shutdown_connections 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/2602528688 >> 172.21.15.45:0/2602528688 conn(0x7fb4b00a2bd0 msgr2=0x7fb4b00a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/2602528688 shutdown_connections 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/2602528688 wait complete. 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient: build_initial_monmap 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient: monmap: 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 librados: starting msgr at 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 librados: starting objecter 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 Processor -- start 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 -- start start 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 librados: setting wanted keys 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 1 librados: calling monclient init 2024-09-16T03:23:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.495+0000 7fb4b7fc6640 10 monclient: init 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b009fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b0135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b013c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b0071250 con 0x7fb4b013c010 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b0071530 con 0x7fb4b0135950 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b0074240 con 0x7fb4b009fc40 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient(hunting): authenticate will time out at 2128.148160s 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b5d62640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb4b009fc40 0x7fb4b006e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58694/0 (socket says 172.21.15.45:58694) 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b6563640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb4b013c010 0x7fb4b01d2280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60498/0 (socket says 172.21.15.45:60498) 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b5d62640 1 -- 172.21.15.45:0/3234966286 learned_addr learned my addr 172.21.15.45:0/3234966286 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 client.?.objecter ms_handle_connect 0x7fb4b013c010 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 client.?.objecter ms_handle_connect 0x7fb4b009fc40 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 961976724 0 0) 0x7fb4b0074240 con 0x7fb4b009fc40 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): my global_id is 184220 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb494004ec0 con 0x7fb4b009fc40 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932746662 0 0) 0x7fb4b0071250 con 0x7fb4b013c010 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): my global_id is 174324 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4b0074240 con 0x7fb4b013c010 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 client.?.objecter ms_handle_connect 0x7fb4b0135950 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3072398385 0 0) 0x7fb494004ec0 con 0x7fb4b009fc40 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4b0071250 con 0x7fb4b009fc40 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb4ac001360 con 0x7fb4b009fc40 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): dump: 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1949193431 0 0) 0x7fb4b0074240 con 0x7fb4b013c010 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb494004ec0 con 0x7fb4b013c010 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8662695 0 0) 0x7fb4b0071530 con 0x7fb4b0135950 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): my global_id is 164407 2024-09-16T03:23:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4b0074240 con 0x7fb4b0135950 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb4a8001f90 con 0x7fb4b013c010 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient(hunting): dump: 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1807001118 0 0) 0x7fb4b0071250 con 0x7fb4b009fc40 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 >> v1:172.21.15.46:6789/0 conn(0x7fb4b0135950 legacy=0x7fb4b01ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 >> v1:172.21.15.45:6789/0 conn(0x7fb4b013c010 legacy=0x7fb4b01d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient: _finish_hunting 0 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 monclient: found mon.c 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b0071f80 con 0x7fb4b009fc40 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient: _finish_auth 0 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 5 monclient: authenticate success, global_id 184220 2024-09-16T03:23:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient: _renew_subs 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b00702f0 con 0x7fb4b009fc40 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 client.184220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient: _renew_subs 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb4b01d3b70 con 0x7fb4b009fc40 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 1 librados: init done 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4ac001620 con 0x7fb4b009fc40 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7fb4ac004c40 con 0x7fb4b009fc40 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 monclient: dump: 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a7fff640 4 set_mon_vals no callback set 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb498005090 con 0x7fb4b009fc40 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7fb4ac010cb0 con 0x7fb4b009fc40 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7fb4ac04b630 con 0x7fb4b009fc40 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 client.184220.objecter ms_dispatch 0x7fb4b0000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 3 client.184220.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 3 client.184220.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 20 client.184220.objecter dump_active .. 0 homeless 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4b5561640 10 monclient: get_auth_request con 0x7fb49403e1f0 auth_method 0 2024-09-16T03:23:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.499+0000 7fb4a67fc640 10 client.184220.objecter ms_handle_connect 0x7fb49403e1f0 2024-09-16T03:23:57.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.503+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fb4ac016bd0 con 0x7fb4b009fc40 2024-09-16T03:23:57.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.503+0000 7fb4a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.503+0000 7fb4a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:57.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.671+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fb498000ba0 con 0x7fb49403e1f0 2024-09-16T03:23:57.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.671+0000 7fb4a67fc640 1 -- 172.21.15.45:0/3234966286 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7fb498000ba0 con 0x7fb49403e1f0 2024-09-16T03:23:57.672 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T03:23:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 10 librados: watch_flush enter 2024-09-16T03:23:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 10 librados: watch_flush exit 2024-09-16T03:23:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 20 client.184220.objecter shutdown clearing up homeless session... 2024-09-16T03:23:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 10 client.184220.objecter successfully canceled tick 2024-09-16T03:23:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 >> v1:172.21.15.46:6812/149360280 conn(0x7fb49403e1f0 legacy=0x7fb4940406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 10 monclient: shutdown 2024-09-16T03:23:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 >> v1:172.21.15.45:6790/0 conn(0x7fb4b009fc40 legacy=0x7fb4b006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 shutdown_connections 2024-09-16T03:23:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 >> 172.21.15.45:0/3234966286 conn(0x7fb4b00a2bd0 msgr2=0x7fb4b013ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 shutdown_connections 2024-09-16T03:23:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 1 -- 172.21.15.45:0/3234966286 wait complete. 2024-09-16T03:23:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.675+0000 7fb4b7fc6640 1 librados: shutdown 2024-09-16T03:23:57.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:23:57.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.807+0000 7f90530d9640 10 monclient: get_monmap_and_config 2024-09-16T03:23:57.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.807+0000 7f90530d9640 10 monclient: build_initial_monmap 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.807+0000 7f90530d9640 10 monclient: monmap: 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:23:57.811847+0000 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:23:57.811847+0000 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:23:57.810 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 Processor -- start 2024-09-16T03:23:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- start start 2024-09-16T03:23:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient: init 2024-09-16T03:23:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:23:57.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient(hunting): picked mon.noname-c con 0x7f904c135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient(hunting): picked mon.noname-b con 0x7f904c09fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient(hunting): picked mon.noname-a con 0x7f904c13c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c09f7e0 con 0x7f904c13c010 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c09b160 con 0x7f904c135950 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c09b440 con 0x7f904c09fc40 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient(hunting): authenticate will time out at 2128.461148s 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f904c09fc40 0x7f904c13a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58700/0 (socket says 172.21.15.45:58700) 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904bfff640 1 -- 172.21.15.45:0/30255811 learned_addr learned my addr 172.21.15.45:0/30255811 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4230696715 0 0) 0x7f904c09b440 con 0x7f904c09fc40 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): my global_id is 184223 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9034003530 con 0x7f904c09fc40 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 671345125 0 0) 0x7f904c09f7e0 con 0x7f904c13c010 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): my global_id is 174327 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f904c09b440 con 0x7f904c13c010 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1547254072 0 0) 0x7f904c09b160 con 0x7f904c135950 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient(hunting): my global_id is 164410 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f904c09f7e0 con 0x7f904c135950 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 877547959 0 0) 0x7f9034003530 con 0x7f904c09fc40 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 >> v1:172.21.15.46:6789/0 conn(0x7f904c135950 legacy=0x7f904c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 >> v1:172.21.15.45:6789/0 conn(0x7f904c13c010 legacy=0x7f904c13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 monclient: found mon.noname-b 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904c136440 con 0x7f904c09fc40 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:23:57.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f903c0016a0 con 0x7f904c09fc40 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: dump: 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 5 monclient: authenticate success, global_id 184223 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f903c001840 con 0x7f904c09fc40 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 1 -- 172.21.15.45:0/30255811 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f903c001c90 con 0x7f904c09fc40 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient: get_monmap_and_config success 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 4 set_mon_vals no callback set 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:57.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f904b7fe640 10 monclient: dump: 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient: shutdown 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- 172.21.15.45:0/30255811 >> v1:172.21.15.45:6790/0 conn(0x7f904c09fc40 legacy=0x7f904c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- 172.21.15.45:0/30255811 shutdown_connections 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- 172.21.15.45:0/30255811 >> 172.21.15.45:0/30255811 conn(0x7f904c0a2bd0 msgr2=0x7f904c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:57.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- 172.21.15.45:0/30255811 shutdown_connections 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- 172.21.15.45:0/30255811 wait complete. 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient: build_initial_monmap 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient: monmap: 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 librados: starting msgr at 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 librados: starting objecter 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 Processor -- start 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 -- start start 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 librados: setting wanted keys 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 1 librados: calling monclient init 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.811+0000 7f90530d9640 10 monclient: init 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient: _reopen_session rank -1 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient(hunting): picked mon.noname-a con 0x7f904c09fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient(hunting): picked mon.noname-c con 0x7f904c135950 addr v1:172.21.15.45:6790/0 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient(hunting): picked mon.noname-b con 0x7f904c13c010 addr v1:172.21.15.46:6789/0 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c09d9d0 con 0x7f904c09fc40 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c13bb20 con 0x7f904c13c010 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c09d2c0 con 0x7f904c135950 2024-09-16T03:23:57.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient(hunting): _renew_subs 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient(hunting): authenticate will time out at 2128.464239s 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f904bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f904c135950 0x7f904c1d1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58714/0 (socket says 172.21.15.45:58714) 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f9050e75640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f904c09fc40 0x7f904c1cd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60512/0 (socket says 172.21.15.45:60512) 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f904bfff640 1 -- 172.21.15.45:0/2663851507 learned_addr learned my addr 172.21.15.45:0/2663851507 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 client.?.objecter ms_handle_connect 0x7f904c09fc40 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 client.?.objecter ms_handle_connect 0x7f904c135950 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717139631 0 0) 0x7f904c09d9d0 con 0x7f904c09fc40 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): my global_id is 174330 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9038004ef0 con 0x7f904c09fc40 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3061418409 0 0) 0x7f904c09d2c0 con 0x7f904c135950 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): my global_id is 184226 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f904c09d9d0 con 0x7f904c135950 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 client.?.objecter ms_handle_connect 0x7f904c13c010 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2731077653 0 0) 0x7f9038004ef0 con 0x7f904c09fc40 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f904c09d2c0 con 0x7f904c09fc40 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f9040001980 con 0x7f904c09fc40 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): dump: 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3818608080 0 0) 0x7f904c09d9d0 con 0x7f904c135950 2024-09-16T03:23:57.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9038004ef0 con 0x7f904c135950 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f903c001b90 con 0x7f904c135950 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): dump: 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.829 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404483545 0 0) 0x7f904c13bb20 con 0x7f904c13c010 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _negotiate 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient(hunting): my global_id is 164413 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f904c09d9d0 con 0x7f904c13c010 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4145691870 0 0) 0x7f9038004ef0 con 0x7f904c135950 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 >> v1:172.21.15.46:6789/0 conn(0x7f904c13c010 legacy=0x7f904c1d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 >> v1:172.21.15.45:6789/0 conn(0x7f904c09fc40 legacy=0x7f904c1cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient: _finish_hunting 0 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 monclient: found mon.c 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:23:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904c09e890 con 0x7f904c135950 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient: _finish_auth 0 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient: _check_auth_tickets 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 5 monclient: authenticate success, global_id 184226 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient: _renew_subs 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f904c134fd0 con 0x7f904c135950 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 client.184226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient: _renew_subs 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f904c1cd7e0 con 0x7f904c135950 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 1 librados: init done 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f903c001180 con 0x7f904c135950 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:23:57.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f903c0017a0 con 0x7f904c135950 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f904affd640 4 set_mon_vals no callback set 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 monclient: dump: 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:23:57.832 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f902c005090 con 0x7f904c135950 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f903c010970 con 0x7f904c135950 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f903c04b280 con 0x7f904c135950 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 client.184226.objecter ms_dispatch 0x7f904c000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 3 client.184226.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 3 client.184226.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 20 client.184226.objecter dump_active .. 0 homeless 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f9050e75640 10 monclient: get_auth_request con 0x7f903803e320 auth_method 0 2024-09-16T03:23:57.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.815+0000 7f90497fa640 10 client.184226.objecter ms_handle_connect 0x7f903803e320 2024-09-16T03:23:57.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.819+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f903c016880 con 0x7f904c135950 2024-09-16T03:23:57.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.819+0000 7f90497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:23:57.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.819+0000 7f90497fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:23:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.987+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f902c000ba0 con 0x7f903803e320 2024-09-16T03:23:57.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.995+0000 7f90497fa640 1 -- 172.21.15.45:0/2663851507 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114047 (unknown 2965378022 0 1698679795) 0x7f902c000ba0 con 0x7f903803e320 2024-09-16T03:23:57.996 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:23:57.996 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-16T03:23:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.995+0000 7f90530d9640 10 librados: watch_flush enter 2024-09-16T03:23:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.995+0000 7f90530d9640 10 librados: watch_flush exit 2024-09-16T03:23:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.995+0000 7f90530d9640 20 client.184226.objecter shutdown clearing up homeless session... 2024-09-16T03:23:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.995+0000 7f90530d9640 10 client.184226.objecter successfully canceled tick 2024-09-16T03:23:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.995+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 >> v1:172.21.15.46:6812/149360280 conn(0x7f903803e320 legacy=0x7f90380407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.995+0000 7f90530d9640 10 monclient: shutdown 2024-09-16T03:23:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.995+0000 7f90530d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:23:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.995+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 >> v1:172.21.15.45:6790/0 conn(0x7f904c135950 legacy=0x7f904c1d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:23:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.999+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 shutdown_connections 2024-09-16T03:23:57.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.999+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 >> 172.21.15.45:0/2663851507 conn(0x7f904c0a2bd0 msgr2=0x7f904c13ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:23:57.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.999+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 shutdown_connections 2024-09-16T03:23:57.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.999+0000 7f90530d9640 1 -- 172.21.15.45:0/2663851507 wait complete. 2024-09-16T03:23:57.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:23:57.999+0000 7f90530d9640 1 librados: shutdown 2024-09-16T03:23:58.014 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1264,"stamp":"2024-09-16T03:23:57.871951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":469008,"kb_used_data":52776,"kb_used_omap":9,"kb_used_meta":416054,"kb_avail":565762032,"statfs":{"total":579820584960,"available":579340320768,"internally_reserved":0,"allocated":54042624,"data_stored":25919678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":426039970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004114"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210772+0000","last_change":"2024-09-16T03:06:02.756614+0000","last_active":"2024-09-16T03:22:27.210772+0000","last_peered":"2024-09-16T03:22:27.210772+0000","last_clean":"2024-09-16T03:22:27.210772+0000","last_became_active":"2024-09-16T03:06:02.756187+0000","last_became_peered":"2024-09-16T03:06:02.756187+0000","last_unstale":"2024-09-16T03:22:27.210772+0000","last_undegraded":"2024-09-16T03:22:27.210772+0000","last_fullsized":"2024-09-16T03:22:27.210772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:43.602727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211989+0000","last_change":"2024-09-16T03:06:02.806920+0000","last_active":"2024-09-16T03:22:27.211989+0000","last_peered":"2024-09-16T03:22:27.211989+0000","last_clean":"2024-09-16T03:22:27.211989+0000","last_became_active":"2024-09-16T03:06:02.798283+0000","last_became_peered":"2024-09-16T03:06:02.798283+0000","last_unstale":"2024-09-16T03:22:27.211989+0000","last_undegraded":"2024-09-16T03:22:27.211989+0000","last_fullsized":"2024-09-16T03:22:27.211989+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:43.018291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.212003+0000","last_change":"2024-09-16T03:06:02.832611+0000","last_active":"2024-09-16T03:22:27.212003+0000","last_peered":"2024-09-16T03:22:27.212003+0000","last_clean":"2024-09-16T03:22:27.212003+0000","last_became_active":"2024-09-16T03:06:02.830437+0000","last_became_peered":"2024-09-16T03:06:02.830437+0000","last_unstale":"2024-09-16T03:22:27.212003+0000","last_undegraded":"2024-09-16T03:22:27.212003+0000","last_fullsized":"2024-09-16T03:22:27.212003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:25.137935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774614+0000","last_change":"2024-09-16T03:06:02.664001+0000","last_active":"2024-09-16T03:22:26.774614+0000","last_peered":"2024-09-16T03:22:26.774614+0000","last_clean":"2024-09-16T03:22:26.774614+0000","last_became_active":"2024-09-16T03:06:02.661502+0000","last_became_peered":"2024-09-16T03:06:02.661502+0000","last_unstale":"2024-09-16T03:22:26.774614+0000","last_undegraded":"2024-09-16T03:22:26.774614+0000","last_fullsized":"2024-09-16T03:22:26.774614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:56.502316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209178+0000","last_change":"2024-09-16T03:06:02.750362+0000","last_active":"2024-09-16T03:22:27.209178+0000","last_peered":"2024-09-16T03:22:27.209178+0000","last_clean":"2024-09-16T03:22:27.209178+0000","last_became_active":"2024-09-16T03:06:02.744921+0000","last_became_peered":"2024-09-16T03:06:02.744921+0000","last_unstale":"2024-09-16T03:22:27.209178+0000","last_undegraded":"2024-09-16T03:22:27.209178+0000","last_fullsized":"2024-09-16T03:22:27.209178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:03.960933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1336,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777944+0000","last_change":"2024-09-16T03:06:02.747722+0000","last_active":"2024-09-16T03:22:26.777944+0000","last_peered":"2024-09-16T03:22:26.777944+0000","last_clean":"2024-09-16T03:22:26.777944+0000","last_became_active":"2024-09-16T03:06:02.746529+0000","last_became_peered":"2024-09-16T03:06:02.746529+0000","last_unstale":"2024-09-16T03:22:26.777944+0000","last_undegraded":"2024-09-16T03:22:26.777944+0000","last_fullsized":"2024-09-16T03:22:26.777944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:54.961886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209716+0000","last_change":"2024-09-16T03:06:02.755623+0000","last_active":"2024-09-16T03:22:27.209716+0000","last_peered":"2024-09-16T03:22:27.209716+0000","last_clean":"2024-09-16T03:22:27.209716+0000","last_became_active":"2024-09-16T03:06:02.755275+0000","last_became_peered":"2024-09-16T03:06:02.755275+0000","last_unstale":"2024-09-16T03:22:27.209716+0000","last_undegraded":"2024-09-16T03:22:27.209716+0000","last_fullsized":"2024-09-16T03:22:27.209716+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:53.461709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122045+0000","last_change":"2024-09-16T03:06:02.820568+0000","last_active":"2024-09-16T03:22:27.122045+0000","last_peered":"2024-09-16T03:22:27.122045+0000","last_clean":"2024-09-16T03:22:27.122045+0000","last_became_active":"2024-09-16T03:06:02.820265+0000","last_became_peered":"2024-09-16T03:06:02.820265+0000","last_unstale":"2024-09-16T03:22:27.122045+0000","last_undegraded":"2024-09-16T03:22:27.122045+0000","last_fullsized":"2024-09-16T03:22:27.122045+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:30.846375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774889+0000","last_change":"2024-09-16T03:06:02.699671+0000","last_active":"2024-09-16T03:22:26.774889+0000","last_peered":"2024-09-16T03:22:26.774889+0000","last_clean":"2024-09-16T03:22:26.774889+0000","last_became_active":"2024-09-16T03:06:02.698531+0000","last_became_peered":"2024-09-16T03:06:02.698531+0000","last_unstale":"2024-09-16T03:22:26.774889+0000","last_undegraded":"2024-09-16T03:22:26.774889+0000","last_fullsized":"2024-09-16T03:22:26.774889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:19:31.192437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777641+0000","last_change":"2024-09-16T03:06:02.829600+0000","last_active":"2024-09-16T03:22:26.777641+0000","last_peered":"2024-09-16T03:22:26.777641+0000","last_clean":"2024-09-16T03:22:26.777641+0000","last_became_active":"2024-09-16T03:06:02.829225+0000","last_became_peered":"2024-09-16T03:06:02.829225+0000","last_unstale":"2024-09-16T03:22:26.777641+0000","last_undegraded":"2024-09-16T03:22:26.777641+0000","last_fullsized":"2024-09-16T03:22:26.777641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:02.011377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211564+0000","last_change":"2024-09-16T03:06:02.755653+0000","last_active":"2024-09-16T03:22:27.211564+0000","last_peered":"2024-09-16T03:22:27.211564+0000","last_clean":"2024-09-16T03:22:27.211564+0000","last_became_active":"2024-09-16T03:06:02.750016+0000","last_became_peered":"2024-09-16T03:06:02.750016+0000","last_unstale":"2024-09-16T03:22:27.211564+0000","last_undegraded":"2024-09-16T03:22:27.211564+0000","last_fullsized":"2024-09-16T03:22:27.211564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:44.526283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777284+0000","last_change":"2024-09-16T03:06:02.858185+0000","last_active":"2024-09-16T03:22:26.777284+0000","last_peered":"2024-09-16T03:22:26.777284+0000","last_clean":"2024-09-16T03:22:26.777284+0000","last_became_active":"2024-09-16T03:06:02.857164+0000","last_became_peered":"2024-09-16T03:06:02.857164+0000","last_unstale":"2024-09-16T03:22:26.777284+0000","last_undegraded":"2024-09-16T03:22:26.777284+0000","last_fullsized":"2024-09-16T03:22:26.777284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:44.538058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1356,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209549+0000","last_change":"2024-09-16T03:04:59.132718+0000","last_active":"2024-09-16T03:22:27.209549+0000","last_peered":"2024-09-16T03:22:27.209549+0000","last_clean":"2024-09-16T03:22:27.209549+0000","last_became_active":"2024-09-16T03:04:57.691813+0000","last_became_peered":"2024-09-16T03:04:57.691813+0000","last_unstale":"2024-09-16T03:22:27.209549+0000","last_undegraded":"2024-09-16T03:22:27.209549+0000","last_fullsized":"2024-09-16T03:22:27.209549+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:55.115695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015712199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210838+0000","last_change":"2024-09-16T03:06:02.721325+0000","last_active":"2024-09-16T03:22:27.210838+0000","last_peered":"2024-09-16T03:22:27.210838+0000","last_clean":"2024-09-16T03:22:27.210838+0000","last_became_active":"2024-09-16T03:06:02.721049+0000","last_became_peered":"2024-09-16T03:06:02.721049+0000","last_unstale":"2024-09-16T03:22:27.210838+0000","last_undegraded":"2024-09-16T03:22:27.210838+0000","last_fullsized":"2024-09-16T03:22:27.210838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:56.615690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1363,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.123007+0000","last_change":"2024-09-16T03:04:59.142318+0000","last_active":"2024-09-16T03:22:27.123007+0000","last_peered":"2024-09-16T03:22:27.123007+0000","last_clean":"2024-09-16T03:22:27.123007+0000","last_became_active":"2024-09-16T03:04:57.117464+0000","last_became_peered":"2024-09-16T03:04:57.117464+0000","last_unstale":"2024-09-16T03:22:27.123007+0000","last_undegraded":"2024-09-16T03:22:27.123007+0000","last_fullsized":"2024-09-16T03:22:27.123007+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:54:40.258398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012748499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209678+0000","last_change":"2024-09-16T03:06:02.769149+0000","last_active":"2024-09-16T03:22:27.209678+0000","last_peered":"2024-09-16T03:22:27.209678+0000","last_clean":"2024-09-16T03:22:27.209678+0000","last_became_active":"2024-09-16T03:06:02.768926+0000","last_became_peered":"2024-09-16T03:06:02.768926+0000","last_unstale":"2024-09-16T03:22:27.209678+0000","last_undegraded":"2024-09-16T03:22:27.209678+0000","last_fullsized":"2024-09-16T03:22:27.209678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:11.356942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1357,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209966+0000","last_change":"2024-09-16T03:04:59.131904+0000","last_active":"2024-09-16T03:22:27.209966+0000","last_peered":"2024-09-16T03:22:27.209966+0000","last_clean":"2024-09-16T03:22:27.209966+0000","last_became_active":"2024-09-16T03:04:57.113180+0000","last_became_peered":"2024-09-16T03:04:57.113180+0000","last_unstale":"2024-09-16T03:22:27.209966+0000","last_undegraded":"2024-09-16T03:22:27.209966+0000","last_fullsized":"2024-09-16T03:22:27.209966+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:30.278408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087237100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1361,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122484+0000","last_change":"2024-09-16T03:06:02.678570+0000","last_active":"2024-09-16T03:22:27.122484+0000","last_peered":"2024-09-16T03:22:27.122484+0000","last_clean":"2024-09-16T03:22:27.122484+0000","last_became_active":"2024-09-16T03:06:02.662210+0000","last_became_peered":"2024-09-16T03:06:02.662210+0000","last_unstale":"2024-09-16T03:22:27.122484+0000","last_undegraded":"2024-09-16T03:22:27.122484+0000","last_fullsized":"2024-09-16T03:22:27.122484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:18.499817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1363,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122893+0000","last_change":"2024-09-16T03:04:59.142273+0000","last_active":"2024-09-16T03:22:27.122893+0000","last_peered":"2024-09-16T03:22:27.122893+0000","last_clean":"2024-09-16T03:22:27.122893+0000","last_became_active":"2024-09-16T03:04:57.117660+0000","last_became_peered":"2024-09-16T03:04:57.117660+0000","last_unstale":"2024-09-16T03:22:27.122893+0000","last_undegraded":"2024-09-16T03:22:27.122893+0000","last_fullsized":"2024-09-16T03:22:27.122893+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:22.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211578+0000","last_change":"2024-09-16T03:06:02.853549+0000","last_active":"2024-09-16T03:22:27.211578+0000","last_peered":"2024-09-16T03:22:27.211578+0000","last_clean":"2024-09-16T03:22:27.211578+0000","last_became_active":"2024-09-16T03:06:02.852993+0000","last_became_peered":"2024-09-16T03:06:02.852993+0000","last_unstale":"2024-09-16T03:22:27.211578+0000","last_undegraded":"2024-09-16T03:22:27.211578+0000","last_fullsized":"2024-09-16T03:22:27.211578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:42.416142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774729+0000","last_change":"2024-09-16T03:04:59.138496+0000","last_active":"2024-09-16T03:22:26.774729+0000","last_peered":"2024-09-16T03:22:26.774729+0000","last_clean":"2024-09-16T03:22:26.774729+0000","last_became_active":"2024-09-16T03:04:57.119695+0000","last_became_peered":"2024-09-16T03:04:57.119695+0000","last_unstale":"2024-09-16T03:22:26.774729+0000","last_undegraded":"2024-09-16T03:22:26.774729+0000","last_fullsized":"2024-09-16T03:22:26.774729+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:24.631318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024209740000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.123006+0000","last_change":"2024-09-16T03:06:02.765932+0000","last_active":"2024-09-16T03:22:27.123006+0000","last_peered":"2024-09-16T03:22:27.123006+0000","last_clean":"2024-09-16T03:22:27.123006+0000","last_became_active":"2024-09-16T03:06:02.760889+0000","last_became_peered":"2024-09-16T03:06:02.760889+0000","last_unstale":"2024-09-16T03:22:27.123006+0000","last_undegraded":"2024-09-16T03:22:27.123006+0000","last_fullsized":"2024-09-16T03:22:27.123006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:26.594180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1352,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777443+0000","last_change":"2024-09-16T03:04:59.133304+0000","last_active":"2024-09-16T03:22:26.777443+0000","last_peered":"2024-09-16T03:22:26.777443+0000","last_clean":"2024-09-16T03:22:26.777443+0000","last_became_active":"2024-09-16T03:04:57.118413+0000","last_became_peered":"2024-09-16T03:04:57.118413+0000","last_unstale":"2024-09-16T03:22:26.777443+0000","last_undegraded":"2024-09-16T03:22:26.777443+0000","last_fullsized":"2024-09-16T03:22:26.777443+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:24.300841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014587720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774981+0000","last_change":"2024-09-16T03:06:02.746071+0000","last_active":"2024-09-16T03:22:26.774981+0000","last_peered":"2024-09-16T03:22:26.774981+0000","last_clean":"2024-09-16T03:22:26.774981+0000","last_became_active":"2024-09-16T03:06:02.743279+0000","last_became_peered":"2024-09-16T03:06:02.743279+0000","last_unstale":"2024-09-16T03:22:26.774981+0000","last_undegraded":"2024-09-16T03:22:26.774981+0000","last_fullsized":"2024-09-16T03:22:26.774981+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:12.479265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1357,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209320+0000","last_change":"2024-09-16T03:04:59.132034+0000","last_active":"2024-09-16T03:22:27.209320+0000","last_peered":"2024-09-16T03:22:27.209320+0000","last_clean":"2024-09-16T03:22:27.209320+0000","last_became_active":"2024-09-16T03:04:57.117048+0000","last_became_peered":"2024-09-16T03:04:57.117048+0000","last_unstale":"2024-09-16T03:22:27.209320+0000","last_undegraded":"2024-09-16T03:22:27.209320+0000","last_fullsized":"2024-09-16T03:22:27.209320+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:57.899748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099232000000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122142+0000","last_change":"2024-09-16T03:06:02.822684+0000","last_active":"2024-09-16T03:22:27.122142+0000","last_peered":"2024-09-16T03:22:27.122142+0000","last_clean":"2024-09-16T03:22:27.122142+0000","last_became_active":"2024-09-16T03:06:02.820554+0000","last_became_peered":"2024-09-16T03:06:02.820554+0000","last_unstale":"2024-09-16T03:22:27.122142+0000","last_undegraded":"2024-09-16T03:22:27.122142+0000","last_fullsized":"2024-09-16T03:22:27.122142+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:24:48.622458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774841+0000","last_change":"2024-09-16T03:04:59.137133+0000","last_active":"2024-09-16T03:22:26.774841+0000","last_peered":"2024-09-16T03:22:26.774841+0000","last_clean":"2024-09-16T03:22:26.774841+0000","last_became_active":"2024-09-16T03:04:57.116849+0000","last_became_peered":"2024-09-16T03:04:57.116849+0000","last_unstale":"2024-09-16T03:22:26.774841+0000","last_undegraded":"2024-09-16T03:22:26.774841+0000","last_fullsized":"2024-09-16T03:22:26.774841+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:56.093368+0000","last_clean_scrub_stamp":"2024-09-16T03:04:56.093368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:51.385856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00103192,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1448,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209486+0000","last_change":"2024-09-16T03:04:51.319467+0000","last_active":"2024-09-16T03:22:27.209486+0000","last_peered":"2024-09-16T03:22:27.209486+0000","last_clean":"2024-09-16T03:22:27.209486+0000","last_became_active":"2024-09-16T03:04:51.317160+0000","last_became_peered":"2024-09-16T03:04:51.317160+0000","last_unstale":"2024-09-16T03:22:27.209486+0000","last_undegraded":"2024-09-16T03:22:27.209486+0000","last_fullsized":"2024-09-16T03:22:27.209486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:04:49.540253+0000","last_clean_scrub_stamp":"2024-09-16T03:04:49.540253+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:15.016218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.775063+0000","last_change":"2024-09-16T03:06:02.762194+0000","last_active":"2024-09-16T03:22:26.775063+0000","last_peered":"2024-09-16T03:22:26.775063+0000","last_clean":"2024-09-16T03:22:26.775063+0000","last_became_active":"2024-09-16T03:06:02.761928+0000","last_became_peered":"2024-09-16T03:06:02.761928+0000","last_unstale":"2024-09-16T03:22:26.775063+0000","last_undegraded":"2024-09-16T03:22:26.775063+0000","last_fullsized":"2024-09-16T03:22:26.775063+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:27:32.863289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774795+0000","last_change":"2024-09-16T03:06:02.664328+0000","last_active":"2024-09-16T03:22:26.774795+0000","last_peered":"2024-09-16T03:22:26.774795+0000","last_clean":"2024-09-16T03:22:26.774795+0000","last_became_active":"2024-09-16T03:06:02.661047+0000","last_became_peered":"2024-09-16T03:06:02.661047+0000","last_unstale":"2024-09-16T03:22:26.774795+0000","last_undegraded":"2024-09-16T03:22:26.774795+0000","last_fullsized":"2024-09-16T03:22:26.774795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:52.055646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1346,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122655+0000","last_change":"2024-09-16T03:06:02.769535+0000","last_active":"2024-09-16T03:22:27.122655+0000","last_peered":"2024-09-16T03:22:27.122655+0000","last_clean":"2024-09-16T03:22:27.122655+0000","last_became_active":"2024-09-16T03:06:02.769042+0000","last_became_peered":"2024-09-16T03:06:02.769042+0000","last_unstale":"2024-09-16T03:22:27.122655+0000","last_undegraded":"2024-09-16T03:22:27.122655+0000","last_fullsized":"2024-09-16T03:22:27.122655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:26.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777928+0000","last_change":"2024-09-16T03:06:02.829591+0000","last_active":"2024-09-16T03:22:26.777928+0000","last_peered":"2024-09-16T03:22:26.777928+0000","last_clean":"2024-09-16T03:22:26.777928+0000","last_became_active":"2024-09-16T03:06:02.829311+0000","last_became_peered":"2024-09-16T03:06:02.829311+0000","last_unstale":"2024-09-16T03:22:26.777928+0000","last_undegraded":"2024-09-16T03:22:26.777928+0000","last_fullsized":"2024-09-16T03:22:26.777928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:04.717263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211030+0000","last_change":"2024-09-16T03:06:02.821360+0000","last_active":"2024-09-16T03:22:27.211030+0000","last_peered":"2024-09-16T03:22:27.211030+0000","last_clean":"2024-09-16T03:22:27.211030+0000","last_became_active":"2024-09-16T03:06:02.820637+0000","last_became_peered":"2024-09-16T03:06:02.820637+0000","last_unstale":"2024-09-16T03:22:27.211030+0000","last_undegraded":"2024-09-16T03:22:27.211030+0000","last_fullsized":"2024-09-16T03:22:27.211030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:56.743641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.211911+0000","last_change":"2024-09-16T03:06:02.800273+0000","last_active":"2024-09-16T03:22:27.211911+0000","last_peered":"2024-09-16T03:22:27.211911+0000","last_clean":"2024-09-16T03:22:27.211911+0000","last_became_active":"2024-09-16T03:06:02.798149+0000","last_became_peered":"2024-09-16T03:06:02.798149+0000","last_unstale":"2024-09-16T03:22:27.211911+0000","last_undegraded":"2024-09-16T03:22:27.211911+0000","last_fullsized":"2024-09-16T03:22:27.211911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:44.563168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209139+0000","last_change":"2024-09-16T03:06:02.739124+0000","last_active":"2024-09-16T03:22:27.209139+0000","last_peered":"2024-09-16T03:22:27.209139+0000","last_clean":"2024-09-16T03:22:27.209139+0000","last_became_active":"2024-09-16T03:06:02.736697+0000","last_became_peered":"2024-09-16T03:06:02.736697+0000","last_unstale":"2024-09-16T03:22:27.209139+0000","last_undegraded":"2024-09-16T03:22:27.209139+0000","last_fullsized":"2024-09-16T03:22:27.209139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:56:28.860331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.122618+0000","last_change":"2024-09-16T03:06:02.661500+0000","last_active":"2024-09-16T03:22:27.122618+0000","last_peered":"2024-09-16T03:22:27.122618+0000","last_clean":"2024-09-16T03:22:27.122618+0000","last_became_active":"2024-09-16T03:06:02.660659+0000","last_became_peered":"2024-09-16T03:06:02.660659+0000","last_unstale":"2024-09-16T03:22:27.122618+0000","last_undegraded":"2024-09-16T03:22:27.122618+0000","last_fullsized":"2024-09-16T03:22:27.122618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:50.530990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.774695+0000","last_change":"2024-09-16T03:06:02.675225+0000","last_active":"2024-09-16T03:22:26.774695+0000","last_peered":"2024-09-16T03:22:26.774695+0000","last_clean":"2024-09-16T03:22:26.774695+0000","last_became_active":"2024-09-16T03:06:02.668156+0000","last_became_peered":"2024-09-16T03:06:02.668156+0000","last_unstale":"2024-09-16T03:22:26.774695+0000","last_undegraded":"2024-09-16T03:22:26.774695+0000","last_fullsized":"2024-09-16T03:22:26.774695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:10:19.961091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.209688+0000","last_change":"2024-09-16T03:06:02.755239+0000","last_active":"2024-09-16T03:22:27.209688+0000","last_peered":"2024-09-16T03:22:27.209688+0000","last_clean":"2024-09-16T03:22:27.209688+0000","last_became_active":"2024-09-16T03:06:02.752964+0000","last_became_peered":"2024-09-16T03:06:02.752964+0000","last_unstale":"2024-09-16T03:22:27.209688+0000","last_undegraded":"2024-09-16T03:22:27.209688+0000","last_fullsized":"2024-09-16T03:22:27.209688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:17.029098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:26.777476+0000","last_change":"2024-09-16T03:06:02.749131+0000","last_active":"2024-09-16T03:22:26.777476+0000","last_peered":"2024-09-16T03:22:26.777476+0000","last_clean":"2024-09-16T03:22:26.777476+0000","last_became_active":"2024-09-16T03:06:02.748671+0000","last_became_peered":"2024-09-16T03:06:02.748671+0000","last_unstale":"2024-09-16T03:22:26.777476+0000","last_undegraded":"2024-09-16T03:22:26.777476+0000","last_fullsized":"2024-09-16T03:22:26.777476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.998002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.210912+0000","last_change":"2024-09-16T03:06:02.723416+0000","last_active":"2024-09-16T03:22:27.210912+0000","last_peered":"2024-09-16T03:22:27.210912+0000","last_clean":"2024-09-16T03:22:27.210912+0000","last_became_active":"2024-09-16T03:06:02.723112+0000","last_became_peered":"2024-09-16T03:06:02.723112+0000","last_unstale":"2024-09-16T03:22:27.210912+0000","last_undegraded":"2024-09-16T03:22:27.210912+0000","last_fullsized":"2024-09-16T03:22:27.210912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:02.720767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:22:27.212250+0000","last_change":"2024-09-16T03:06:02.845346+0000","last_active":"2024-09-16T03:22:27.212250+0000","last_peered":"2024-09-16T03:22:27.212250+0000","last_clean":"2024-09-16T03:22:27.212250+0000","last_became_active":"2024-09-16T03:06:02.843866+0000","last_became_peered":"2024-09-16T03:06:02.843866+0000","last_unstale":"2024-09-16T03:22:27.212250+0000","last_undegraded":"2024-09-16T03:22:27.212250+0000","last_fullsized":"2024-09-16T03:22:27.212250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:06:01.583781+0000","last_clean_scrub_stamp":"2024-09-16T03:06:01.583781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:45.180460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607783,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84240,"kb_used_data":8624,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287600,"statfs":{"total":96636764160,"available":96550502400,"internally_reserved":0,"allocated":8830976,"data_stored":4143648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607784,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71748,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300092,"statfs":{"total":96636764160,"available":96563294208,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607783,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84752,"kb_used_data":9144,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287088,"statfs":{"total":96636764160,"available":96549978112,"internally_reserved":0,"allocated":9363456,"data_stored":4672562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607783,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71764,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300076,"statfs":{"total":96636764160,"available":96563277824,"internally_reserved":0,"allocated":8835072,"data_stored":4143653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640488,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72276,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94299564,"statfs":{"total":96636764160,"available":96562753536,"internally_reserved":0,"allocated":9359360,"data_stored":4672557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640489,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84228,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287612,"statfs":{"total":96636764160,"available":96550514688,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:23:58.019 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.019 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.019 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.019 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.019 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.020 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.020 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.020 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.020 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.020 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.020 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.021 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.021 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T03:04:56.093368+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=4, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.021 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.021 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T03:04:56.093368+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=4, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.021 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.021 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T03:04:56.093368+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=4, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.021 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.021 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T03:04:56.093368+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=4, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.022 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.022 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T03:04:56.093368+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=4, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.022 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.022 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T03:04:56.093368+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=4, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.022 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.022 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T03:04:56.093368+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=4, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.022 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.022 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T03:04:56.093368+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=4, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.022 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T03:04:49.540253+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=4, tm_sec=49, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.023 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.024 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.024 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.024 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-09-16T03:06:01.583781+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=6, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=23, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:23:58.024 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T03:24:18.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:24:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.154+0000 7f467d993640 10 monclient: get_monmap_and_config 2024-09-16T03:24:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.154+0000 7f467d993640 10 monclient: build_initial_monmap 2024-09-16T03:24:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.154+0000 7f467d993640 10 monclient: monmap: 2024-09-16T03:24:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:24:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:24:18.159771+0000 2024-09-16T03:24:18.158 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:24:18.159771+0000 2024-09-16T03:24:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:24:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:24:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:24:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:24:18.159 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 Processor -- start 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 -- start start 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient: init 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient: _reopen_session rank -1 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient(hunting): picked mon.noname-a con 0x7f4678135950 addr v1:172.21.15.45:6789/0 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient(hunting): picked mon.noname-c con 0x7f467809fc40 addr v1:172.21.15.46:6789/0 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient(hunting): picked mon.noname-b con 0x7f467813c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467809f7e0 con 0x7f4678135950 2024-09-16T03:24:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467809b160 con 0x7f467809fc40 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467809b440 con 0x7f467813c010 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient(hunting): _renew_subs 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient(hunting): authenticate will time out at 2148.809194s 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f4676ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4678135950 0x7f4678135d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42504/0 (socket says 172.21.15.45:42504) 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f4676ffd640 1 -- 172.21.15.45:0/3354175544 learned_addr learned my addr 172.21.15.45:0/3354175544 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46777fe640 1 --1- 172.21.15.45:0/3354175544 >> v1:172.21.15.45:6790/0 conn(0x7f467813c010 0x7f467813e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:41006/0 (socket says 172.21.15.45:41006) 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 656825963 0 0) 0x7f467809b440 con 0x7f467813c010 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient(hunting): my global_id is 184229 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4664003530 con 0x7f467813c010 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3549331507 0 0) 0x7f467809f7e0 con 0x7f4678135950 2024-09-16T03:24:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient(hunting): my global_id is 174333 2024-09-16T03:24:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f467809b440 con 0x7f4678135950 2024-09-16T03:24:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 688897655 0 0) 0x7f467809b440 con 0x7f4678135950 2024-09-16T03:24:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 >> v1:172.21.15.45:6790/0 conn(0x7f467813c010 legacy=0x7f467813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 >> v1:172.21.15.46:6789/0 conn(0x7f467809fc40 legacy=0x7f467813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: _finish_hunting 0 2024-09-16T03:24:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 monclient: found mon.noname-a 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4678136440 con 0x7f4678135950 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: _finish_auth 0 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: _check_auth_tickets 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4668001680 con 0x7f4678135950 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 5 monclient: authenticate success, global_id 174333 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:24:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: dump: 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.171 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:24:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4668001820 con 0x7f4678135950 2024-09-16T03:24:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:24:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 1 -- 172.21.15.45:0/3354175544 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4668001cb0 con 0x7f4678135950 2024-09-16T03:24:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient: get_monmap_and_config success 2024-09-16T03:24:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 4 set_mon_vals no callback set 2024-09-16T03:24:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:24:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:24:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f46767fc640 10 monclient: dump: 2024-09-16T03:24:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient: shutdown 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 -- 172.21.15.45:0/3354175544 >> v1:172.21.15.45:6789/0 conn(0x7f4678135950 legacy=0x7f4678135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 -- 172.21.15.45:0/3354175544 shutdown_connections 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 -- 172.21.15.45:0/3354175544 >> 172.21.15.45:0/3354175544 conn(0x7f46780a2bd0 msgr2=0x7f46780a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:24:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 -- 172.21.15.45:0/3354175544 shutdown_connections 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 -- 172.21.15.45:0/3354175544 wait complete. 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient: build_initial_monmap 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 10 monclient: monmap: 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 librados: starting msgr at 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 librados: starting objecter 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.158+0000 7f467d993640 1 Processor -- start 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 -- start start 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 librados: setting wanted keys 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 librados: calling monclient init 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient: init 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient: _reopen_session rank -1 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient(hunting): picked mon.noname-a con 0x7f467809fc40 addr v1:172.21.15.45:6789/0 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient(hunting): picked mon.noname-b con 0x7f4678135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient(hunting): picked mon.noname-c con 0x7f467813c010 addr v1:172.21.15.45:6790/0 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467806e4f0 con 0x7f467809fc40 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467806f740 con 0x7f4678135950 2024-09-16T03:24:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4678074120 con 0x7f467813c010 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient(hunting): _renew_subs 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient(hunting): authenticate will time out at 2148.812597s 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4676ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f467809fc40 0x7f46781392a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42520/0 (socket says 172.21.15.45:42520) 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4676ffd640 1 -- 172.21.15.45:0/3292383846 learned_addr learned my addr 172.21.15.45:0/3292383846 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 client.?.objecter ms_handle_connect 0x7f467809fc40 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 client.?.objecter ms_handle_connect 0x7f467813c010 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84789469 0 0) 0x7f4678074120 con 0x7f467813c010 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): my global_id is 184232 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4658004ec0 con 0x7f467813c010 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 345441270 0 0) 0x7f467806e4f0 con 0x7f467809fc40 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): my global_id is 174336 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4678074120 con 0x7f467809fc40 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 client.?.objecter ms_handle_connect 0x7f4678135950 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 475989090 0 0) 0x7f4658004ec0 con 0x7f467813c010 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f467806e4f0 con 0x7f467813c010 2024-09-16T03:24:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f46700016a0 con 0x7f467813c010 2024-09-16T03:24:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:24:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:24:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): dump: 2024-09-16T03:24:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1859435832 0 0) 0x7f4678074120 con 0x7f467809fc40 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4658004ec0 con 0x7f467809fc40 2024-09-16T03:24:18.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4668001bf0 con 0x7f467809fc40 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): dump: 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.181 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2177812013 0 0) 0x7f467806f740 con 0x7f4678135950 2024-09-16T03:24:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient(hunting): my global_id is 164419 2024-09-16T03:24:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4678074120 con 0x7f4678135950 2024-09-16T03:24:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1582527704 0 0) 0x7f467806e4f0 con 0x7f467813c010 2024-09-16T03:24:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 >> v1:172.21.15.46:6789/0 conn(0x7f4678135950 legacy=0x7f46781399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 >> v1:172.21.15.45:6789/0 conn(0x7f467809fc40 legacy=0x7f46781392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient: _finish_hunting 0 2024-09-16T03:24:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 monclient: found mon.c 2024-09-16T03:24:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467813f740 con 0x7f467813c010 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient: _finish_auth 0 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 5 monclient: authenticate success, global_id 184232 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient: _renew_subs 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f467813b110 con 0x7f467813c010 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 client.184232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient: _renew_subs 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:24:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f467813a0c0 con 0x7f467813c010 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 librados: init done 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4670004b70 con 0x7f467813c010 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f4670004ec0 con 0x7f467813c010 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 monclient: dump: 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4675ffb640 4 set_mon_vals no callback set 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f465c005090 con 0x7f467813c010 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f4670010f30 con 0x7f467813c010 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f467004b690 con 0x7f467813c010 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 client.184232.objecter ms_dispatch 0x7f4678000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:24:18.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 3 client.184232.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:24:18.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 3 client.184232.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:24:18.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 20 client.184232.objecter dump_active .. 0 homeless 2024-09-16T03:24:18.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4676ffd640 10 monclient: get_auth_request con 0x7f465803e1a0 auth_method 0 2024-09-16T03:24:18.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.162+0000 7f4674ff9640 10 client.184232.objecter ms_handle_connect 0x7f465803e1a0 2024-09-16T03:24:18.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.166+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4670016720 con 0x7f467813c010 2024-09-16T03:24:18.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.166+0000 7f4674ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:24:18.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.166+0000 7f4674ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:24:18.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.338+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 --> v1:172.21.15.46:6812/149360280 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f465c000ba0 con 0x7f465803e1a0 2024-09-16T03:24:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.342+0000 7f4674ff9640 1 -- 172.21.15.45:0/3292383846 <== mgr.4101 v1:172.21.15.46:6812/149360280 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114069 (unknown 2965378022 0 2168667027) 0x7f465c000ba0 con 0x7f465803e1a0 2024-09-16T03:24:18.345 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-16T03:24:18.345 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-16T03:24:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 10 librados: watch_flush enter 2024-09-16T03:24:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 10 librados: watch_flush exit 2024-09-16T03:24:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 20 client.184232.objecter shutdown clearing up homeless session... 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 10 client.184232.objecter successfully canceled tick 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 >> v1:172.21.15.46:6812/149360280 conn(0x7f465803e1a0 legacy=0x7f4658040660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 10 monclient: shutdown 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 >> v1:172.21.15.45:6790/0 conn(0x7f467813c010 legacy=0x7f46781d2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 shutdown_connections 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 >> 172.21.15.45:0/3292383846 conn(0x7f46780a2bd0 msgr2=0x7f4678134e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 shutdown_connections 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 1 -- 172.21.15.45:0/3292383846 wait complete. 2024-09-16T03:24:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.346+0000 7f467d993640 1 librados: shutdown 2024-09-16T03:24:18.365 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1274,"stamp":"2024-09-16T03:24:17.878886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":468936,"kb_used_data":52776,"kb_used_omap":9,"kb_used_meta":416054,"kb_avail":565762104,"statfs":{"total":579820584960,"available":579340394496,"internally_reserved":0,"allocated":54042624,"data_stored":25919678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":426039970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004292"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:58.556497+0000","last_change":"2024-09-16T03:23:58.556497+0000","last_active":"2024-09-16T03:23:58.556497+0000","last_peered":"2024-09-16T03:23:58.556497+0000","last_clean":"2024-09-16T03:23:58.556497+0000","last_became_active":"2024-09-16T03:06:02.756187+0000","last_became_peered":"2024-09-16T03:06:02.756187+0000","last_unstale":"2024-09-16T03:23:58.556497+0000","last_undegraded":"2024-09-16T03:23:58.556497+0000","last_fullsized":"2024-09-16T03:23:58.556497+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:58.556308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:58.556308+0000","last_clean_scrub_stamp":"2024-09-16T03:23:58.556308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:21.102122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:00.798495+0000","last_change":"2024-09-16T03:24:00.798495+0000","last_active":"2024-09-16T03:24:00.798495+0000","last_peered":"2024-09-16T03:24:00.798495+0000","last_clean":"2024-09-16T03:24:00.798495+0000","last_became_active":"2024-09-16T03:06:02.798283+0000","last_became_peered":"2024-09-16T03:06:02.798283+0000","last_unstale":"2024-09-16T03:24:00.798495+0000","last_undegraded":"2024-09-16T03:24:00.798495+0000","last_fullsized":"2024-09-16T03:24:00.798495+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:00.798338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:00.798338+0000","last_clean_scrub_stamp":"2024-09-16T03:24:00.798338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:33:16.953880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:59.759369+0000","last_change":"2024-09-16T03:23:59.759369+0000","last_active":"2024-09-16T03:23:59.759369+0000","last_peered":"2024-09-16T03:23:59.759369+0000","last_clean":"2024-09-16T03:23:59.759369+0000","last_became_active":"2024-09-16T03:06:02.830437+0000","last_became_peered":"2024-09-16T03:06:02.830437+0000","last_unstale":"2024-09-16T03:23:59.759369+0000","last_undegraded":"2024-09-16T03:23:59.759369+0000","last_fullsized":"2024-09-16T03:23:59.759369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:59.759199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:59.759199+0000","last_clean_scrub_stamp":"2024-09-16T03:23:59.759199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:02.369974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:05.159447+0000","last_change":"2024-09-16T03:24:05.159447+0000","last_active":"2024-09-16T03:24:05.159447+0000","last_peered":"2024-09-16T03:24:05.159447+0000","last_clean":"2024-09-16T03:24:05.159447+0000","last_became_active":"2024-09-16T03:06:02.661502+0000","last_became_peered":"2024-09-16T03:06:02.661502+0000","last_unstale":"2024-09-16T03:24:05.159447+0000","last_undegraded":"2024-09-16T03:24:05.159447+0000","last_fullsized":"2024-09-16T03:24:05.159447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:05.159283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:05.159283+0000","last_clean_scrub_stamp":"2024-09-16T03:24:05.159283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:40:06.735255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:04.939395+0000","last_change":"2024-09-16T03:24:04.939395+0000","last_active":"2024-09-16T03:24:04.939395+0000","last_peered":"2024-09-16T03:24:04.939395+0000","last_clean":"2024-09-16T03:24:04.939395+0000","last_became_active":"2024-09-16T03:06:02.744921+0000","last_became_peered":"2024-09-16T03:06:02.744921+0000","last_unstale":"2024-09-16T03:24:04.939395+0000","last_undegraded":"2024-09-16T03:24:04.939395+0000","last_fullsized":"2024-09-16T03:24:04.939395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:04.939231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:04.939231+0000","last_clean_scrub_stamp":"2024-09-16T03:24:04.939231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:18.818243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1344,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:01.293487+0000","last_change":"2024-09-16T03:24:01.293487+0000","last_active":"2024-09-16T03:24:01.293487+0000","last_peered":"2024-09-16T03:24:01.293487+0000","last_clean":"2024-09-16T03:24:01.293487+0000","last_became_active":"2024-09-16T03:06:02.746529+0000","last_became_peered":"2024-09-16T03:06:02.746529+0000","last_unstale":"2024-09-16T03:24:01.293487+0000","last_undegraded":"2024-09-16T03:24:01.293487+0000","last_fullsized":"2024-09-16T03:24:01.293487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:01.293323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:01.293323+0000","last_clean_scrub_stamp":"2024-09-16T03:24:01.293323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:57.966163+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:02.927932+0000","last_change":"2024-09-16T03:24:02.927932+0000","last_active":"2024-09-16T03:24:02.927932+0000","last_peered":"2024-09-16T03:24:02.927932+0000","last_clean":"2024-09-16T03:24:02.927932+0000","last_became_active":"2024-09-16T03:06:02.755275+0000","last_became_peered":"2024-09-16T03:06:02.755275+0000","last_unstale":"2024-09-16T03:24:02.927932+0000","last_undegraded":"2024-09-16T03:24:02.927932+0000","last_fullsized":"2024-09-16T03:24:02.927932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:02.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:02.927774+0000","last_clean_scrub_stamp":"2024-09-16T03:24:02.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:31:06.804658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1355,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:02.981952+0000","last_change":"2024-09-16T03:24:02.981952+0000","last_active":"2024-09-16T03:24:02.981952+0000","last_peered":"2024-09-16T03:24:02.981952+0000","last_clean":"2024-09-16T03:24:02.981952+0000","last_became_active":"2024-09-16T03:06:02.820265+0000","last_became_peered":"2024-09-16T03:06:02.820265+0000","last_unstale":"2024-09-16T03:24:02.981952+0000","last_undegraded":"2024-09-16T03:24:02.981952+0000","last_fullsized":"2024-09-16T03:24:02.981952+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:02.981778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:02.981778+0000","last_clean_scrub_stamp":"2024-09-16T03:24:02.981778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:23.285822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:04.133066+0000","last_change":"2024-09-16T03:24:04.133066+0000","last_active":"2024-09-16T03:24:04.133066+0000","last_peered":"2024-09-16T03:24:04.133066+0000","last_clean":"2024-09-16T03:24:04.133066+0000","last_became_active":"2024-09-16T03:06:02.698531+0000","last_became_peered":"2024-09-16T03:06:02.698531+0000","last_unstale":"2024-09-16T03:24:04.133066+0000","last_undegraded":"2024-09-16T03:24:04.133066+0000","last_fullsized":"2024-09-16T03:24:04.133066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:04.132870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:04.132870+0000","last_clean_scrub_stamp":"2024-09-16T03:24:04.132870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:30.495697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:58.295384+0000","last_change":"2024-09-16T03:23:58.295384+0000","last_active":"2024-09-16T03:23:58.295384+0000","last_peered":"2024-09-16T03:23:58.295384+0000","last_clean":"2024-09-16T03:23:58.295384+0000","last_became_active":"2024-09-16T03:06:02.829225+0000","last_became_peered":"2024-09-16T03:06:02.829225+0000","last_unstale":"2024-09-16T03:23:58.295384+0000","last_undegraded":"2024-09-16T03:23:58.295384+0000","last_fullsized":"2024-09-16T03:23:58.295384+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:58.295220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:58.295220+0000","last_clean_scrub_stamp":"2024-09-16T03:23:58.295220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:01:43.967790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:58.757506+0000","last_change":"2024-09-16T03:23:58.757506+0000","last_active":"2024-09-16T03:23:58.757506+0000","last_peered":"2024-09-16T03:23:58.757506+0000","last_clean":"2024-09-16T03:23:58.757506+0000","last_became_active":"2024-09-16T03:06:02.750016+0000","last_became_peered":"2024-09-16T03:06:02.750016+0000","last_unstale":"2024-09-16T03:23:58.757506+0000","last_undegraded":"2024-09-16T03:23:58.757506+0000","last_fullsized":"2024-09-16T03:23:58.757506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:58.757345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:58.757345+0000","last_clean_scrub_stamp":"2024-09-16T03:23:58.757345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:10.870907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:59.261327+0000","last_change":"2024-09-16T03:23:59.261327+0000","last_active":"2024-09-16T03:23:59.261327+0000","last_peered":"2024-09-16T03:23:59.261327+0000","last_clean":"2024-09-16T03:23:59.261327+0000","last_became_active":"2024-09-16T03:06:02.857164+0000","last_became_peered":"2024-09-16T03:06:02.857164+0000","last_unstale":"2024-09-16T03:23:59.261327+0000","last_undegraded":"2024-09-16T03:23:59.261327+0000","last_fullsized":"2024-09-16T03:23:59.261327+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:59.261152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:59.261152+0000","last_clean_scrub_stamp":"2024-09-16T03:23:59.261152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:18.191594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1364,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:57.949584+0000","last_change":"2024-09-16T03:23:57.949584+0000","last_active":"2024-09-16T03:23:57.949584+0000","last_peered":"2024-09-16T03:23:57.949584+0000","last_clean":"2024-09-16T03:23:57.949584+0000","last_became_active":"2024-09-16T03:04:57.691813+0000","last_became_peered":"2024-09-16T03:04:57.691813+0000","last_unstale":"2024-09-16T03:23:57.949584+0000","last_undegraded":"2024-09-16T03:23:57.949584+0000","last_fullsized":"2024-09-16T03:23:57.949584+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:57.949427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:57.949427+0000","last_clean_scrub_stamp":"2024-09-16T03:23:57.949427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:14:57.474135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015712199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1342,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:59.512280+0000","last_change":"2024-09-16T03:23:59.512280+0000","last_active":"2024-09-16T03:23:59.512280+0000","last_peered":"2024-09-16T03:23:59.512280+0000","last_clean":"2024-09-16T03:23:59.512280+0000","last_became_active":"2024-09-16T03:06:02.721049+0000","last_became_peered":"2024-09-16T03:06:02.721049+0000","last_unstale":"2024-09-16T03:23:59.512280+0000","last_undegraded":"2024-09-16T03:23:59.512280+0000","last_fullsized":"2024-09-16T03:23:59.512280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:59.512119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:59.512119+0000","last_clean_scrub_stamp":"2024-09-16T03:23:59.512119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:36.741977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1371,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:57.112470+0000","last_change":"2024-09-16T03:23:57.112470+0000","last_active":"2024-09-16T03:23:57.112470+0000","last_peered":"2024-09-16T03:23:57.112470+0000","last_clean":"2024-09-16T03:23:57.112470+0000","last_became_active":"2024-09-16T03:04:57.117464+0000","last_became_peered":"2024-09-16T03:04:57.117464+0000","last_unstale":"2024-09-16T03:23:57.112470+0000","last_undegraded":"2024-09-16T03:23:57.112470+0000","last_fullsized":"2024-09-16T03:23:57.112470+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:57.112367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:57.112367+0000","last_clean_scrub_stamp":"2024-09-16T03:23:57.112367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:23.588673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012748499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:58.951033+0000","last_change":"2024-09-16T03:23:58.951033+0000","last_active":"2024-09-16T03:23:58.951033+0000","last_peered":"2024-09-16T03:23:58.951033+0000","last_clean":"2024-09-16T03:23:58.951033+0000","last_became_active":"2024-09-16T03:06:02.768926+0000","last_became_peered":"2024-09-16T03:06:02.768926+0000","last_unstale":"2024-09-16T03:23:58.951033+0000","last_undegraded":"2024-09-16T03:23:58.951033+0000","last_fullsized":"2024-09-16T03:23:58.951033+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:58.950863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:58.950863+0000","last_clean_scrub_stamp":"2024-09-16T03:23:58.950863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:07.938011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1365,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:01.958899+0000","last_change":"2024-09-16T03:24:01.958899+0000","last_active":"2024-09-16T03:24:01.958899+0000","last_peered":"2024-09-16T03:24:01.958899+0000","last_clean":"2024-09-16T03:24:01.958899+0000","last_became_active":"2024-09-16T03:04:57.113180+0000","last_became_peered":"2024-09-16T03:04:57.113180+0000","last_unstale":"2024-09-16T03:24:01.958899+0000","last_undegraded":"2024-09-16T03:24:01.958899+0000","last_fullsized":"2024-09-16T03:24:01.958899+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:01.958707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:01.958707+0000","last_clean_scrub_stamp":"2024-09-16T03:24:01.958707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:16.179875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087237100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"703'8","reported_seq":1370,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:58.098586+0000","last_change":"2024-09-16T03:23:58.098586+0000","last_active":"2024-09-16T03:23:58.098586+0000","last_peered":"2024-09-16T03:23:58.098586+0000","last_clean":"2024-09-16T03:23:58.098586+0000","last_became_active":"2024-09-16T03:06:02.662210+0000","last_became_peered":"2024-09-16T03:06:02.662210+0000","last_unstale":"2024-09-16T03:23:58.098586+0000","last_undegraded":"2024-09-16T03:23:58.098586+0000","last_fullsized":"2024-09-16T03:23:58.098586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'8","last_scrub_stamp":"2024-09-16T03:23:58.098411+0000","last_deep_scrub":"703'8","last_deep_scrub_stamp":"2024-09-16T03:23:58.098411+0000","last_clean_scrub_stamp":"2024-09-16T03:23:58.098411+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:20.025312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1371,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:56.105945+0000","last_change":"2024-09-16T03:23:56.105945+0000","last_active":"2024-09-16T03:23:56.105945+0000","last_peered":"2024-09-16T03:23:56.105945+0000","last_clean":"2024-09-16T03:23:56.105945+0000","last_became_active":"2024-09-16T03:04:57.117660+0000","last_became_peered":"2024-09-16T03:04:57.117660+0000","last_unstale":"2024-09-16T03:23:56.105945+0000","last_undegraded":"2024-09-16T03:23:56.105945+0000","last_fullsized":"2024-09-16T03:23:56.105945+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:56.105781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:56.105781+0000","last_clean_scrub_stamp":"2024-09-16T03:23:56.105781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:50:49.048182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001291344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:56.720023+0000","last_change":"2024-09-16T03:23:56.719973+0000","last_active":"2024-09-16T03:23:56.720023+0000","last_peered":"2024-09-16T03:23:56.720023+0000","last_clean":"2024-09-16T03:23:56.720023+0000","last_became_active":"2024-09-16T03:06:02.852993+0000","last_became_peered":"2024-09-16T03:06:02.852993+0000","last_unstale":"2024-09-16T03:23:56.720023+0000","last_undegraded":"2024-09-16T03:23:56.720023+0000","last_fullsized":"2024-09-16T03:23:56.720023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:56.719900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:56.719900+0000","last_clean_scrub_stamp":"2024-09-16T03:23:56.719900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:30.840123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"703'3","reported_seq":1368,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:58.040622+0000","last_change":"2024-09-16T03:23:58.040622+0000","last_active":"2024-09-16T03:23:58.040622+0000","last_peered":"2024-09-16T03:23:58.040622+0000","last_clean":"2024-09-16T03:23:58.040622+0000","last_became_active":"2024-09-16T03:04:57.119695+0000","last_became_peered":"2024-09-16T03:04:57.119695+0000","last_unstale":"2024-09-16T03:23:58.040622+0000","last_undegraded":"2024-09-16T03:23:58.040622+0000","last_fullsized":"2024-09-16T03:23:58.040622+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'3","last_scrub_stamp":"2024-09-16T03:23:58.040424+0000","last_deep_scrub":"703'3","last_deep_scrub_stamp":"2024-09-16T03:23:58.040424+0000","last_clean_scrub_stamp":"2024-09-16T03:23:58.040424+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:35:37.274804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024209740000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1356,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:00.040753+0000","last_change":"2024-09-16T03:24:00.040753+0000","last_active":"2024-09-16T03:24:00.040753+0000","last_peered":"2024-09-16T03:24:00.040753+0000","last_clean":"2024-09-16T03:24:00.040753+0000","last_became_active":"2024-09-16T03:06:02.760889+0000","last_became_peered":"2024-09-16T03:06:02.760889+0000","last_unstale":"2024-09-16T03:24:00.040753+0000","last_undegraded":"2024-09-16T03:24:00.040753+0000","last_fullsized":"2024-09-16T03:24:00.040753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:00.040586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:00.040586+0000","last_clean_scrub_stamp":"2024-09-16T03:24:00.040586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:43.398171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1360,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:57.268461+0000","last_change":"2024-09-16T03:23:57.268461+0000","last_active":"2024-09-16T03:23:57.268461+0000","last_peered":"2024-09-16T03:23:57.268461+0000","last_clean":"2024-09-16T03:23:57.268461+0000","last_became_active":"2024-09-16T03:04:57.118413+0000","last_became_peered":"2024-09-16T03:04:57.118413+0000","last_unstale":"2024-09-16T03:23:57.268461+0000","last_undegraded":"2024-09-16T03:23:57.268461+0000","last_fullsized":"2024-09-16T03:23:57.268461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:57.268340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:57.268340+0000","last_clean_scrub_stamp":"2024-09-16T03:23:57.268340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:53.944346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014587720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:00.073884+0000","last_change":"2024-09-16T03:24:00.073884+0000","last_active":"2024-09-16T03:24:00.073884+0000","last_peered":"2024-09-16T03:24:00.073884+0000","last_clean":"2024-09-16T03:24:00.073884+0000","last_became_active":"2024-09-16T03:06:02.743279+0000","last_became_peered":"2024-09-16T03:06:02.743279+0000","last_unstale":"2024-09-16T03:24:00.073884+0000","last_undegraded":"2024-09-16T03:24:00.073884+0000","last_fullsized":"2024-09-16T03:24:00.073884+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:00.073701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:00.073701+0000","last_clean_scrub_stamp":"2024-09-16T03:24:00.073701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:51:57.470332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1365,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:01.004313+0000","last_change":"2024-09-16T03:24:01.004313+0000","last_active":"2024-09-16T03:24:01.004313+0000","last_peered":"2024-09-16T03:24:01.004313+0000","last_clean":"2024-09-16T03:24:01.004313+0000","last_became_active":"2024-09-16T03:04:57.117048+0000","last_became_peered":"2024-09-16T03:04:57.117048+0000","last_unstale":"2024-09-16T03:24:01.004313+0000","last_undegraded":"2024-09-16T03:24:01.004313+0000","last_fullsized":"2024-09-16T03:24:01.004313+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:01.004150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:01.004150+0000","last_clean_scrub_stamp":"2024-09-16T03:24:01.004150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:51.178539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099232000000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1355,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:59.080739+0000","last_change":"2024-09-16T03:23:59.080739+0000","last_active":"2024-09-16T03:23:59.080739+0000","last_peered":"2024-09-16T03:23:59.080739+0000","last_clean":"2024-09-16T03:23:59.080739+0000","last_became_active":"2024-09-16T03:06:02.820554+0000","last_became_peered":"2024-09-16T03:06:02.820554+0000","last_unstale":"2024-09-16T03:23:59.080739+0000","last_undegraded":"2024-09-16T03:23:59.080739+0000","last_fullsized":"2024-09-16T03:23:59.080739+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:59.080570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:59.080570+0000","last_clean_scrub_stamp":"2024-09-16T03:23:59.080570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:08.804348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"703'2","reported_seq":1368,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:01.059279+0000","last_change":"2024-09-16T03:24:01.059279+0000","last_active":"2024-09-16T03:24:01.059279+0000","last_peered":"2024-09-16T03:24:01.059279+0000","last_clean":"2024-09-16T03:24:01.059279+0000","last_became_active":"2024-09-16T03:04:57.116849+0000","last_became_peered":"2024-09-16T03:04:57.116849+0000","last_unstale":"2024-09-16T03:24:01.059279+0000","last_undegraded":"2024-09-16T03:24:01.059279+0000","last_fullsized":"2024-09-16T03:24:01.059279+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'2","last_scrub_stamp":"2024-09-16T03:24:01.059071+0000","last_deep_scrub":"703'2","last_deep_scrub_stamp":"2024-09-16T03:24:01.059071+0000","last_clean_scrub_stamp":"2024-09-16T03:24:01.059071+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:45:14.116583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00103192,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"703'74","reported_seq":1458,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:56.949149+0000","last_change":"2024-09-16T03:23:56.949149+0000","last_active":"2024-09-16T03:23:56.949149+0000","last_peered":"2024-09-16T03:23:56.949149+0000","last_clean":"2024-09-16T03:23:56.949149+0000","last_became_active":"2024-09-16T03:04:51.317160+0000","last_became_peered":"2024-09-16T03:04:51.317160+0000","last_unstale":"2024-09-16T03:23:56.949149+0000","last_undegraded":"2024-09-16T03:23:56.949149+0000","last_fullsized":"2024-09-16T03:23:56.949149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'74","last_scrub_stamp":"2024-09-16T03:23:56.948982+0000","last_deep_scrub":"703'74","last_deep_scrub_stamp":"2024-09-16T03:23:56.948982+0000","last_clean_scrub_stamp":"2024-09-16T03:23:56.948982+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:01.445869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:59.035419+0000","last_change":"2024-09-16T03:23:59.035419+0000","last_active":"2024-09-16T03:23:59.035419+0000","last_peered":"2024-09-16T03:23:59.035419+0000","last_clean":"2024-09-16T03:23:59.035419+0000","last_became_active":"2024-09-16T03:06:02.761928+0000","last_became_peered":"2024-09-16T03:06:02.761928+0000","last_unstale":"2024-09-16T03:23:59.035419+0000","last_undegraded":"2024-09-16T03:23:59.035419+0000","last_fullsized":"2024-09-16T03:23:59.035419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:59.035226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:59.035226+0000","last_clean_scrub_stamp":"2024-09-16T03:23:59.035226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:20.919252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:03.092106+0000","last_change":"2024-09-16T03:24:03.091978+0000","last_active":"2024-09-16T03:24:03.092106+0000","last_peered":"2024-09-16T03:24:03.092106+0000","last_clean":"2024-09-16T03:24:03.092106+0000","last_became_active":"2024-09-16T03:06:02.661047+0000","last_became_peered":"2024-09-16T03:06:02.661047+0000","last_unstale":"2024-09-16T03:24:03.092106+0000","last_undegraded":"2024-09-16T03:24:03.092106+0000","last_fullsized":"2024-09-16T03:24:03.092106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:03.091821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:03.091821+0000","last_clean_scrub_stamp":"2024-09-16T03:24:03.091821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:03.466044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1355,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:01.991982+0000","last_change":"2024-09-16T03:24:01.991757+0000","last_active":"2024-09-16T03:24:01.991982+0000","last_peered":"2024-09-16T03:24:01.991982+0000","last_clean":"2024-09-16T03:24:01.991982+0000","last_became_active":"2024-09-16T03:06:02.769042+0000","last_became_peered":"2024-09-16T03:06:02.769042+0000","last_unstale":"2024-09-16T03:24:01.991982+0000","last_undegraded":"2024-09-16T03:24:01.991982+0000","last_fullsized":"2024-09-16T03:24:01.991982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:01.991568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:01.991568+0000","last_clean_scrub_stamp":"2024-09-16T03:24:01.991568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:26.531654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1344,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:56.298475+0000","last_change":"2024-09-16T03:23:56.298425+0000","last_active":"2024-09-16T03:23:56.298475+0000","last_peered":"2024-09-16T03:23:56.298475+0000","last_clean":"2024-09-16T03:23:56.298475+0000","last_became_active":"2024-09-16T03:06:02.829311+0000","last_became_peered":"2024-09-16T03:06:02.829311+0000","last_unstale":"2024-09-16T03:23:56.298475+0000","last_undegraded":"2024-09-16T03:23:56.298475+0000","last_fullsized":"2024-09-16T03:23:56.298475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:56.298358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:56.298358+0000","last_clean_scrub_stamp":"2024-09-16T03:23:56.298358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:11:21.481592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1342,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:00.478992+0000","last_change":"2024-09-16T03:24:00.478857+0000","last_active":"2024-09-16T03:24:00.478992+0000","last_peered":"2024-09-16T03:24:00.478992+0000","last_clean":"2024-09-16T03:24:00.478992+0000","last_became_active":"2024-09-16T03:06:02.820637+0000","last_became_peered":"2024-09-16T03:06:02.820637+0000","last_unstale":"2024-09-16T03:24:00.478992+0000","last_undegraded":"2024-09-16T03:24:00.478992+0000","last_fullsized":"2024-09-16T03:24:00.478992+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:00.478697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:00.478697+0000","last_clean_scrub_stamp":"2024-09-16T03:24:00.478697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:40.915397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:55.769062+0000","last_change":"2024-09-16T03:23:55.769062+0000","last_active":"2024-09-16T03:23:55.769062+0000","last_peered":"2024-09-16T03:23:55.769062+0000","last_clean":"2024-09-16T03:23:55.769062+0000","last_became_active":"2024-09-16T03:06:02.798149+0000","last_became_peered":"2024-09-16T03:06:02.798149+0000","last_unstale":"2024-09-16T03:23:55.769062+0000","last_undegraded":"2024-09-16T03:23:55.769062+0000","last_fullsized":"2024-09-16T03:23:55.769062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:55.768965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:55.768965+0000","last_clean_scrub_stamp":"2024-09-16T03:23:55.768965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:12.612943+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:59.968631+0000","last_change":"2024-09-16T03:23:59.968631+0000","last_active":"2024-09-16T03:23:59.968631+0000","last_peered":"2024-09-16T03:23:59.968631+0000","last_clean":"2024-09-16T03:23:59.968631+0000","last_became_active":"2024-09-16T03:06:02.736697+0000","last_became_peered":"2024-09-16T03:06:02.736697+0000","last_unstale":"2024-09-16T03:23:59.968631+0000","last_undegraded":"2024-09-16T03:23:59.968631+0000","last_fullsized":"2024-09-16T03:23:59.968631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:59.968444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:59.968444+0000","last_clean_scrub_stamp":"2024-09-16T03:23:59.968444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:28.654626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1356,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:00.992014+0000","last_change":"2024-09-16T03:24:00.991871+0000","last_active":"2024-09-16T03:24:00.992014+0000","last_peered":"2024-09-16T03:24:00.992014+0000","last_clean":"2024-09-16T03:24:00.992014+0000","last_became_active":"2024-09-16T03:06:02.660659+0000","last_became_peered":"2024-09-16T03:06:02.660659+0000","last_unstale":"2024-09-16T03:24:00.992014+0000","last_undegraded":"2024-09-16T03:24:00.992014+0000","last_fullsized":"2024-09-16T03:24:00.992014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:00.991697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:00.991697+0000","last_clean_scrub_stamp":"2024-09-16T03:24:00.991697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:41.071343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:02.075727+0000","last_change":"2024-09-16T03:24:02.075727+0000","last_active":"2024-09-16T03:24:02.075727+0000","last_peered":"2024-09-16T03:24:02.075727+0000","last_clean":"2024-09-16T03:24:02.075727+0000","last_became_active":"2024-09-16T03:06:02.668156+0000","last_became_peered":"2024-09-16T03:06:02.668156+0000","last_unstale":"2024-09-16T03:24:02.075727+0000","last_undegraded":"2024-09-16T03:24:02.075727+0000","last_fullsized":"2024-09-16T03:24:02.075727+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:02.075569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:02.075569+0000","last_clean_scrub_stamp":"2024-09-16T03:24:02.075569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:40:27.973163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:03.934705+0000","last_change":"2024-09-16T03:24:03.934705+0000","last_active":"2024-09-16T03:24:03.934705+0000","last_peered":"2024-09-16T03:24:03.934705+0000","last_clean":"2024-09-16T03:24:03.934705+0000","last_became_active":"2024-09-16T03:06:02.752964+0000","last_became_peered":"2024-09-16T03:06:02.752964+0000","last_unstale":"2024-09-16T03:24:03.934705+0000","last_undegraded":"2024-09-16T03:24:03.934705+0000","last_fullsized":"2024-09-16T03:24:03.934705+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:03.934552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:03.934552+0000","last_clean_scrub_stamp":"2024-09-16T03:24:03.934552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:36.129477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:24:00.293053+0000","last_change":"2024-09-16T03:24:00.293053+0000","last_active":"2024-09-16T03:24:00.293053+0000","last_peered":"2024-09-16T03:24:00.293053+0000","last_clean":"2024-09-16T03:24:00.293053+0000","last_became_active":"2024-09-16T03:06:02.748671+0000","last_became_peered":"2024-09-16T03:06:02.748671+0000","last_unstale":"2024-09-16T03:24:00.293053+0000","last_undegraded":"2024-09-16T03:24:00.293053+0000","last_fullsized":"2024-09-16T03:24:00.293053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:24:00.292895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:24:00.292895+0000","last_clean_scrub_stamp":"2024-09-16T03:24:00.292895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:06.291348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:57.595542+0000","last_change":"2024-09-16T03:23:57.595542+0000","last_active":"2024-09-16T03:23:57.595542+0000","last_peered":"2024-09-16T03:23:57.595542+0000","last_clean":"2024-09-16T03:23:57.595542+0000","last_became_active":"2024-09-16T03:06:02.723112+0000","last_became_peered":"2024-09-16T03:06:02.723112+0000","last_unstale":"2024-09-16T03:23:57.595542+0000","last_undegraded":"2024-09-16T03:23:57.595542+0000","last_fullsized":"2024-09-16T03:23:57.595542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:57.595380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:57.595380+0000","last_clean_scrub_stamp":"2024-09-16T03:23:57.595380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:01.750513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1347,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-16T03:23:57.748106+0000","last_change":"2024-09-16T03:23:57.748106+0000","last_active":"2024-09-16T03:23:57.748106+0000","last_peered":"2024-09-16T03:23:57.748106+0000","last_clean":"2024-09-16T03:23:57.748106+0000","last_became_active":"2024-09-16T03:06:02.843866+0000","last_became_peered":"2024-09-16T03:06:02.843866+0000","last_unstale":"2024-09-16T03:23:57.748106+0000","last_undegraded":"2024-09-16T03:23:57.748106+0000","last_fullsized":"2024-09-16T03:23:57.748106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:23:57.748042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:23:57.748042+0000","last_clean_scrub_stamp":"2024-09-16T03:23:57.748042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:08.929605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607787,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84224,"kb_used_data":8624,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287616,"statfs":{"total":96636764160,"available":96550518784,"internally_reserved":0,"allocated":8830976,"data_stored":4143648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607788,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71740,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300100,"statfs":{"total":96636764160,"available":96563302400,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607787,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84744,"kb_used_data":9144,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287096,"statfs":{"total":96636764160,"available":96549986304,"internally_reserved":0,"allocated":9363456,"data_stored":4672562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607787,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71748,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300092,"statfs":{"total":96636764160,"available":96563294208,"internally_reserved":0,"allocated":8835072,"data_stored":4143653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640492,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72260,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94299580,"statfs":{"total":96636764160,"available":96562769920,"internally_reserved":0,"allocated":9359360,"data_stored":4672557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640493,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":8620,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":8826880,"data_stored":4143629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:24:18.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T03:24:18.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient: get_monmap_and_config 2024-09-16T03:24:18.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient: build_initial_monmap 2024-09-16T03:24:18.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient: monmap: 2024-09-16T03:24:18.486 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:24:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:24:18.488663+0000 2024-09-16T03:24:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:24:18.488663+0000 2024-09-16T03:24:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:24:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:24:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-c 2024-09-16T03:24:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-09-16T03:24:18.487 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 1 Processor -- start 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 1 -- start start 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient: init 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient: _reopen_session rank -1 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient(hunting): picked mon.noname-c con 0x7f00d4135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient(hunting): picked mon.noname-b con 0x7f00d409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient(hunting): picked mon.noname-a con 0x7f00d413c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d409f7e0 con 0x7f00d413c010 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d409b160 con 0x7f00d4135950 2024-09-16T03:24:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d409b440 con 0x7f00d409fc40 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient(hunting): _renew_subs 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient(hunting): authenticate will time out at 2149.138777s 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00da2a3640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f00d413c010 0x7f00d413e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42522/0 (socket says 172.21.15.45:42522) 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00da2a3640 1 -- 172.21.15.45:0/3380543864 learned_addr learned my addr 172.21.15.45:0/3380543864 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d92a1640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f00d409fc40 0x7f00d413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:41024/0 (socket says 172.21.15.45:41024) 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3959855526 0 0) 0x7f00d409f7e0 con 0x7f00d413c010 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient(hunting): my global_id is 174339 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00c0003530 con 0x7f00d413c010 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3249618138 0 0) 0x7f00d409b440 con 0x7f00d409fc40 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient(hunting): my global_id is 184235 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00d409f7e0 con 0x7f00d409fc40 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4152144233 0 0) 0x7f00c0003530 con 0x7f00d413c010 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 >> v1:172.21.15.45:6790/0 conn(0x7f00d409fc40 legacy=0x7f00d413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 >> v1:172.21.15.46:6789/0 conn(0x7f00d4135950 legacy=0x7f00d4135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: _finish_hunting 0 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 monclient: found mon.noname-a 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d4136440 con 0x7f00d413c010 2024-09-16T03:24:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: _finish_auth 0 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: _check_auth_tickets 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f00cc001a20 con 0x7f00d413c010 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: dump: 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.495 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 5 monclient: authenticate success, global_id 174339 2024-09-16T03:24:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:24:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f00cc001bc0 con 0x7f00d413c010 2024-09-16T03:24:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: handle_config config(0 keys) 2024-09-16T03:24:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 1 -- 172.21.15.45:0/3380543864 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f00cc002050 con 0x7f00d413c010 2024-09-16T03:24:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:24:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00d8aa0640 10 monclient: dump: 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 10 monclient: get_monmap_and_config success 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.486+0000 7f00dbd06640 4 set_mon_vals no callback set 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: shutdown 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- 172.21.15.45:0/3380543864 >> v1:172.21.15.45:6789/0 conn(0x7f00d413c010 legacy=0x7f00d413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- 172.21.15.45:0/3380543864 shutdown_connections 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- 172.21.15.45:0/3380543864 >> 172.21.15.45:0/3380543864 conn(0x7f00d40a2bd0 msgr2=0x7f00d40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- 172.21.15.45:0/3380543864 shutdown_connections 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- 172.21.15.45:0/3380543864 wait complete. 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: build_initial_monmap 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: monmap: 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:24:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.noname-b 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 librados: starting msgr at 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 librados: starting objecter 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 Processor -- start 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- start start 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 librados: setting wanted keys 2024-09-16T03:24:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 librados: calling monclient init 2024-09-16T03:24:18.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: init 2024-09-16T03:24:18.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: _reopen_session rank -1 2024-09-16T03:24:18.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:24:18.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient(hunting): picked mon.noname-c con 0x7f00d409fc40 addr v1:172.21.15.45:6790/0 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient(hunting): picked mon.noname-b con 0x7f00d4135950 addr v1:172.21.15.46:6789/0 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient(hunting): picked mon.noname-a con 0x7f00d413c010 addr v1:172.21.15.45:6789/0 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d409f560 con 0x7f00d413c010 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d409d2c0 con 0x7f00d4135950 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d413bb20 con 0x7f00d409fc40 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient(hunting): _renew_subs 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient(hunting): authenticate will time out at 2149.141944s 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00da2a3640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f00d413c010 0x7f00d41d4fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42534/0 (socket says 172.21.15.45:42534) 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00da2a3640 1 -- 172.21.15.45:0/2615812802 learned_addr learned my addr 172.21.15.45:0/2615812802 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 client.?.objecter ms_handle_connect 0x7f00d409fc40 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 client.?.objecter ms_handle_connect 0x7f00d413c010 2024-09-16T03:24:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817775691 0 0) 0x7f00d413bb20 con 0x7f00d409fc40 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): my global_id is 184238 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00b8004ec0 con 0x7f00d409fc40 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 472396684 0 0) 0x7f00d409f560 con 0x7f00d413c010 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): my global_id is 174342 2024-09-16T03:24:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00d413bb20 con 0x7f00d413c010 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 client.?.objecter ms_handle_connect 0x7f00d4135950 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4137861952 0 0) 0x7f00d413bb20 con 0x7f00d413c010 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f00d409f560 con 0x7f00d413c010 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f00cc001f10 con 0x7f00d413c010 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): dump: 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2733150198 0 0) 0x7f00b8004ec0 con 0x7f00d409fc40 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f00d413bb20 con 0x7f00d409fc40 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f00d00018c0 con 0x7f00d409fc40 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): dump: 2024-09-16T03:24:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.1 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3405018583 0 0) 0x7f00d409f7e0 con 0x7f00d4135950 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:24:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient(hunting): my global_id is 164425 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00b8004ec0 con 0x7f00d4135950 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3141977693 0 0) 0x7f00d409f560 con 0x7f00d413c010 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 >> v1:172.21.15.45:6790/0 conn(0x7f00d409fc40 legacy=0x7f00d4099290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 >> v1:172.21.15.46:6789/0 conn(0x7f00d4135950 legacy=0x7f00d40737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 monclient: found mon.a 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d413b6f0 con 0x7f00d413c010 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient: _finish_auth 0 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:24:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 5 monclient: authenticate success, global_id 174342 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: _renew_subs 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f00d409db30 con 0x7f00d413c010 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 client.174342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: _renew_subs 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f00d4099b70 con 0x7f00d413c010 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 librados: init done 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f00cc001ba0 con 0x7f00d413c010 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:24:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3948037039 0 0) 0x7f00cc005120 con 0x7f00d413c010 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00cbfff640 4 set_mon_vals no callback set 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00ca7fc640 10 monclient: dump: 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 16b8bfd8-03c6-4432-90b7-8e9932b37929 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-09-16T03:04:14.884162+0000 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.46:6789/0 mon.b 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-09-16T03:24:18.510 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.490+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00bc005090 con 0x7f00d413c010 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.494+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 2699427206 0 0) 0x7f00cc011190 con 0x7f00d413c010 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.494+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(703..703 src has 99..703) ==== 4685+0+0 (unknown 1423350510 0 0) 0x7f00cc04b750 con 0x7f00d413c010 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.494+0000 7f00ca7fc640 10 client.174342.objecter ms_dispatch 0x7f00d4000ba0 osd_map(703..703 src has 99..703) 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.494+0000 7f00ca7fc640 3 client.174342.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.494+0000 7f00ca7fc640 3 client.174342.objecter handle_osd_map decoding full epoch 703 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.494+0000 7f00ca7fc640 20 client.174342.objecter dump_active .. 0 homeless 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.494+0000 7f00d9aa2640 10 monclient: get_auth_request con 0x7f00b803e2e0 auth_method 0 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.494+0000 7f00ca7fc640 10 client.174342.objecter ms_handle_connect 0x7f00b803e2e0 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.498+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f00cc016760 con 0x7f00d413c010 2024-09-16T03:24:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.498+0000 7f00ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:24:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.498+0000 7f00ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:24:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.666+0000 7f00dbd06640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T03:24:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.666+0000 7f00dbd06640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T03:24:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.666+0000 7f00dbd06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-09-16T03:24:18.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.666+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 --> v1:172.21.15.45:6789/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f00bc005380 con 0x7f00d413c010 2024-09-16T03:24:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.670+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) ==== 123+0+0 (unknown 3414686427 0 0) 0x7f00cc0490f0 con 0x7f00d413c010 2024-09-16T03:24:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.670+0000 7f00ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T03:24:18.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.670+0000 7f00ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:24:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.670+0000 7f00ca7fc640 1 -- 172.21.15.45:0/2615812802 <== mon.0 v1:172.21.15.45:6789/0 11 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7f00cc0162d0 con 0x7f00d413c010 2024-09-16T03:24:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.670+0000 7f00ca7fc640 10 monclient: handle_config config(1 keys) 2024-09-16T03:24:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.670+0000 7f00cb7fe640 4 set_mon_vals no callback set 2024-09-16T03:24:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.670+0000 7f00cb7fe640 10 set_mon_vals mon_health_to_clog = false 2024-09-16T03:24:18.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 10 librados: watch_flush enter 2024-09-16T03:24:18.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 10 librados: watch_flush exit 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 20 client.174342.objecter shutdown clearing up homeless session... 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 10 client.174342.objecter successfully canceled tick 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 >> v1:172.21.15.46:6812/149360280 conn(0x7f00b803e2e0 legacy=0x7f00b80407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 10 monclient: shutdown 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 >> v1:172.21.15.45:6789/0 conn(0x7f00d413c010 legacy=0x7f00d41d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 shutdown_connections 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 >> 172.21.15.45:0/2615812802 conn(0x7f00d40a2bd0 msgr2=0x7f00d413f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 shutdown_connections 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 1 -- 172.21.15.45:0/2615812802 wait complete. 2024-09-16T03:24:18.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-16T03:24:18.674+0000 7f00dbd06640 1 librados: shutdown 2024-09-16T03:24:18.690 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T03:24:18.690 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T03:24:18.690 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T03:24:18.690 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:18.758 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T03:24:18.758 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T03:24:18.758 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:18.827 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T03:24:18.828 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T03:24:18.828 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:19.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-16T03:24:19.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-16T03:24:24.929 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T03:24:24.929 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T03:24:24.930 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:25.007 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T03:24:25.007 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T03:24:25.007 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:25.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~6s 2024-09-16T03:24:25.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-16T03:24:30.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-16T03:24:30.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-16T03:24:31.109 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T03:24:31.110 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T03:24:31.110 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:31.178 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T03:24:31.179 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T03:24:31.179 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T03:24:31.179 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:31.217 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T03:24:31.218 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T03:24:31.218 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:24:31.218 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:31.239 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:24:31.239 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:24:31.239 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:31.299 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:24:31.300 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:24:31.300 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:24:31.328 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:24:31.329 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T03:24:31.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-16T03:24:31.362 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-16T03:24:31.362 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T03:24:31.619 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-16T03:24:31.620 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T03:24:31.767 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-16T03:24:31.767 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T03:24:32.042 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T03:24:32.043 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T03:24:32.333 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T03:24:32.333 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T03:24:32.548 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T03:24:32.549 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T03:24:32.727 INFO:tasks.ceph:Archiving mon data... 2024-09-16T03:24:32.729 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823/data/mon.a.tgz 2024-09-16T03:24:32.729 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-09-16T03:24:32.735 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.mRayr9VghK 2024-09-16T03:24:32.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.mRayr9VghK 2024-09-16T03:24:32.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.mRayr9VghK 2024-09-16T03:24:33.010 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.mRayr9VghK is 750KB 2024-09-16T03:24:33.076 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.mRayr9VghK 2024-09-16T03:24:33.081 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823/data/mon.c.tgz 2024-09-16T03:24:33.082 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-09-16T03:24:33.130 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.wRGXFhvGzZ 2024-09-16T03:24:33.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.wRGXFhvGzZ 2024-09-16T03:24:33.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.wRGXFhvGzZ 2024-09-16T03:24:33.398 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.wRGXFhvGzZ is 749KB 2024-09-16T03:24:33.460 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.wRGXFhvGzZ 2024-09-16T03:24:33.466 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823/data/mon.b.tgz 2024-09-16T03:24:33.466 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-09-16T03:24:33.472 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.A6kCIw8B60 2024-09-16T03:24:33.472 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.A6kCIw8B60 2024-09-16T03:24:33.672 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.A6kCIw8B60 2024-09-16T03:24:33.736 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.A6kCIw8B60 is 753KB 2024-09-16T03:24:33.802 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.A6kCIw8B60 2024-09-16T03:24:33.807 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T03:24:33.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:24:33.810 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:24:33.871 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:24:33.913 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:24:33.913 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T03:24:33.915 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823/remote/smithi045/crash 2024-09-16T03:24:33.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:24:33.930 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823/remote/smithi046/crash 2024-09-16T03:24:33.931 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:24:33.969 INFO:tasks.ceph:Compressing logs... 2024-09-16T03:24:33.969 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:24:33.975 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:24:33.994 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33304.log 2024-09-16T03:24:33.995 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30998.log 2024-09-16T03:24:33.995 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33460.log 2024-09-16T03:24:33.996 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33304.log.gz 2024-09-16T03:24:33.996 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25170.log 2024-09-16T03:24:33.996 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33460.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30998.log.gz 2024-09-16T03:24:33.996 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29474.log 2024-09-16T03:24:33.997 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31305.log 2024-09-16T03:24:33.997 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.25170.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33460.log.gz 2024-09-16T03:24:33.997 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30920.log 2024-09-16T03:24:33.999 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29474.log.gz 2024-09-16T03:24:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33772.log 2024-09-16T03:24:34.000 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.25170.log.gz 2024-09-16T03:24:34.000 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33900.log 2024-09-16T03:24:34.000 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30920.log: /var/log/ceph/ceph-client.admin.33772.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31305.log.gz 2024-09-16T03:24:34.000 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32227.log 2024-09-16T03:24:34.000 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.33772.log.gz 2024-09-16T03:24:34.000 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.30920.log.gz 2024-09-16T03:24:34.000 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33538.log 2024-09-16T03:24:34.000 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28409.log 2024-09-16T03:24:34.000 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33900.log.gz 2024-09-16T03:24:34.002 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32227.log.gz 2024-09-16T03:24:34.002 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30232.log 2024-09-16T03:24:34.002 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33538.log: /var/log/ceph/ceph-client.admin.28409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32718.log 2024-09-16T03:24:34.002 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.33538.log.gz 2024-09-16T03:24:34.002 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32069.log 2024-09-16T03:24:34.002 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30232.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28409.log.gz 2024-09-16T03:24:34.003 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29082.log 2024-09-16T03:24:34.003 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30232.log.gz 2024-09-16T03:24:34.003 INFO:teuthology.orchestra.run.smithi045.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.32718.log.gz 2024-09-16T03:24:34.003 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32997.log 2024-09-16T03:24:34.003 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32069.log: /var/log/ceph/ceph-client.admin.29082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30176.log 2024-09-16T03:24:34.003 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.32069.log.gz 2024-09-16T03:24:34.003 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32305.log 2024-09-16T03:24:34.003 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32997.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29082.log.gz 2024-09-16T03:24:34.005 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30176.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T03:24:34.005 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32997.log.gz 2024-09-16T03:24:34.005 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30176.log.gz 2024-09-16T03:24:34.005 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28429.log 2024-09-16T03:24:34.005 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30972.log 2024-09-16T03:24:34.005 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32305.log.gz 2024-09-16T03:24:34.006 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29764.log 2024-09-16T03:24:34.006 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28429.log: /var/log/ceph/ceph-client.admin.30972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31968.log 2024-09-16T03:24:34.006 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28429.log.gz 2024-09-16T03:24:34.006 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30972.log.gz 2024-09-16T03:24:34.006 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31383.log 2024-09-16T03:24:34.006 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/ceph-client.admin.31968.log: --verbose -- /var/log/ceph/ceph-client.admin.33616.log 2024-09-16T03:24:34.006 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29764.log.gz 2024-09-16T03:24:34.008 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28708.log 2024-09-16T03:24:34.008 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31968.log.gz 2024-09-16T03:24:34.008 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31383.log.gz 2024-09-16T03:24:34.008 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31760.log 2024-09-16T03:24:34.008 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ceph-client.admin.28708.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29108.log 2024-09-16T03:24:34.008 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.33616.log.gz 2024-09-16T03:24:34.008 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28455.log 2024-09-16T03:24:34.008 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28708.log.gz 2024-09-16T03:24:34.009 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.31760.log.gz 2024-09-16T03:24:34.009 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28544.log 2024-09-16T03:24:34.009 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32640.log 2024-09-16T03:24:34.009 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28455.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29108.log.gz 2024-09-16T03:24:34.011 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28908.log 2024-09-16T03:24:34.011 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28455.log.gz 2024-09-16T03:24:34.011 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32640.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28544.log.gz 2024-09-16T03:24:34.011 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31409.log 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31024.log 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32640.log.gz 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31409.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28908.log.gz 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29212.log 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31024.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31409.log.gz 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29290.log 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31024.log.gz 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29342.log 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29212.log.gz 2024-09-16T03:24:34.012 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25097.log 2024-09-16T03:24:34.015 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33382.log 2024-09-16T03:24:34.015 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29342.log.gz 2024-09-16T03:24:34.015 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29290.log.gz 2024-09-16T03:24:34.015 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28384.log 2024-09-16T03:24:34.015 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.25097.log: /var/log/ceph/ceph-client.admin.33382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32917.log 2024-09-16T03:24:34.015 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25097.log.gz 2024-09-16T03:24:34.015 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28384.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33382.log.gz 2024-09-16T03:24:34.016 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32534.log 2024-09-16T03:24:34.016 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ceph-client.admin.32917.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30868.log 2024-09-16T03:24:34.016 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28384.log.gz 2024-09-16T03:24:34.016 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28848.log 2024-09-16T03:24:34.016 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32917.log.gz 2024-09-16T03:24:34.016 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T03:24:34.016 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30868.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32534.log.gz 2024-09-16T03:24:34.016 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32969.log 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30868.log.gz 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29238.log 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28848.log.gz 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28860.log 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33049.log 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32969.log.gz 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30382.log 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29238.log.gz 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31812.log 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.28860.log.gz 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29848.log 2024-09-16T03:24:34.020 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33049.log.gz 2024-09-16T03:24:34.021 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33356.log 2024-09-16T03:24:34.021 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30382.log.gz 2024-09-16T03:24:34.021 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29848.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31812.log.gz 2024-09-16T03:24:34.021 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29526.log 2024-09-16T03:24:34.021 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29848.log.gz 2024-09-16T03:24:34.021 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33075.log 2024-09-16T03:24:34.021 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33356.log: /var/log/ceph/ceph-client.admin.29526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24980.log 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33075.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33356.log.gz 88.4% 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29526.log.gz 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32508.log 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.24980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31565.log 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33075.log.gz 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.24980.log.gz 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29916.log 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29686.log 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32508.log.gz 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29552.log 2024-09-16T03:24:34.026 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31565.log.gz 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30408.log 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29916.log.gz 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32614.log 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29686.log.gz 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33746.log 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29552.log.gz 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31994.log 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30408.log.gz 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32614.log: /var/log/ceph/ceph-client.admin.33746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31890.log 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30842.log 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32614.log.gz 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33746.log.gz 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32943.log 2024-09-16T03:24:34.027 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31994.log.gz 2024-09-16T03:24:34.028 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30842.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31890.log.gz 2024-09-16T03:24:34.028 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28693.log 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30842.log.gz 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30694.log 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29582.log 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28693.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32943.log.gz 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32770.log 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.28693.log.gz 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33876.log 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30694.log.gz 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29582.log: /var/log/ceph/ceph-client.admin.32770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31539.log 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29582.log.gz 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28884.log 2024-09-16T03:24:34.032 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32770.log.gz 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ceph-client.admin.31539.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28551.log 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33876.log.gz 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29160.log 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31539.log.gz 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28551.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28884.log.gz 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31513.log 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29316.log 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28551.log.gz 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28852.log 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.29160.log.gz 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29316.log: gzip 88.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.31513.log.gz -- 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr: /var/log/ceph/ceph-client.admin.28859.log 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33434.log 2024-09-16T03:24:34.033 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29316.log.gz 2024-09-16T03:24:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32666.log 2024-09-16T03:24:34.034 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28852.log.gz 2024-09-16T03:24:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31864.log 2024-09-16T03:24:34.039 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28859.log.gz 2024-09-16T03:24:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31076.log 2024-09-16T03:24:34.039 INFO:teuthology.orchestra.run.smithi045.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.33434.log.gz 2024-09-16T03:24:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33798.log 2024-09-16T03:24:34.039 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32666.log.gz 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33408.log 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31864.log.gz 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33694.log 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.31076.log.gz 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28481.log 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33798.log.gz 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29134.log 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33408.log.gz 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29608.log 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33694.log.gz 2024-09-16T03:24:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31461.log 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28481.log.gz 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32456.log 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29134.log.gz 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29446.log 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31461.log: /var/log/ceph/ceph-client.admin.32456.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33642.log 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29608.log.gz 86.2% -- replaced with /var/log/ceph/ceph-client.admin.31461.log.gz 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30750.log 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32456.log.gz 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29942.log 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33642.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.29446.log.gz 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30750.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33642.log.gz 2024-09-16T03:24:34.041 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33590.log 2024-09-16T03:24:34.042 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.30750.log.gz 2024-09-16T03:24:34.042 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29030.log 2024-09-16T03:24:34.042 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29942.log.gz 2024-09-16T03:24:34.042 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32562.log 2024-09-16T03:24:34.047 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29790.log 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33590.log.gz 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29500.log 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29030.log.gz 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29790.log: gzip -5 --verbose -- 87.2% /var/log/ceph/ceph-client.admin.31186.log 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32562.log.gz 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29500.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31786.log 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29790.log.gz 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30564.log 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29500.log.gz 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31786.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31186.log.gz 2024-09-16T03:24:34.048 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31786.log.gz 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30316.log 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33128.log 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30316.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30894.log 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30564.log.gz 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30590.log 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.30316.log.gz 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31279.log 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.33128.log.gz 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28359.log 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.30894.log.gz 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30590.log: /var/log/ceph/ceph-client.admin.31279.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.30590.log.gz 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28707.log 2024-09-16T03:24:34.049 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31279.log.gz 2024-09-16T03:24:34.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32095.log 2024-09-16T03:24:34.050 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28359.log.gz 2024-09-16T03:24:34.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31050.log 2024-09-16T03:24:34.050 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ceph-client.admin.32095.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30434.log 2024-09-16T03:24:34.050 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28707.log.gz 2024-09-16T03:24:34.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32692.log 2024-09-16T03:24:34.050 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32095.log.gz 2024-09-16T03:24:34.050 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.31050.log.gz 2024-09-16T03:24:34.061 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29919.log 2024-09-16T03:24:34.061 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25150.log 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29902.log 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.25150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29508.log 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29902.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.25150.log.gz 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29918.log 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.29919.log.gz 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28628.log 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29799.log 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28628.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.29918.log.gz 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28514.log 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29152.log 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25174.log 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29152.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29902.log.gz 2024-09-16T03:24:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28851.log 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.25174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29142.log 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3%/var/log/ceph/ceph-client.admin.28851.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.29799.log.gz 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28506.log 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25174.log.gz 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29927.log 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29142.log: /var/log/ceph/ceph-client.admin.28506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29009.log 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.29142.log.gz 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25102.log 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1%/var/log/ceph/ceph-client.admin.29009.log: -- replaced with /var/log/ceph/ceph-client.admin.29152.log.gz 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29927.log.gz 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28750.log 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.25102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28543.log 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28750.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25102.log.gz 2024-09-16T03:24:34.063 INFO:teuthology.orchestra.run.smithi046.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.28514.log.gz 2024-09-16T03:24:34.064 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29924.log 2024-09-16T03:24:34.064 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29917.log 2024-09-16T03:24:34.077 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29924.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.28851.log.gz 2024-09-16T03:24:34.077 INFO:teuthology.orchestra.run.smithi046.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.admin.29009.log.gz 2024-09-16T03:24:34.077 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29262.log 2024-09-16T03:24:34.077 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29917.log: 91.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.29924.log.gz 2024-09-16T03:24:34.077 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29153.log 2024-09-16T03:24:34.077 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28673.log 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28543.log.gz/var/log/ceph/ceph-client.admin.29262.log: 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29153.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29508.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29541.log 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.29153.log.gz 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.29262.log.gz 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28949.log 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29141.log 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29917.log.gz/var/log/ceph/ceph-client.admin.28949.log: 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25126.log 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29294.log 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29141.log: /var/log/ceph/ceph-client.admin.25126.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.25126.log.gz 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29905.log 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29294.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28750.log.gz 2024-09-16T03:24:34.078 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29141.log.gz 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30236.log 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29905.log: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28673.log.gz 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29001.log 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.3.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.28949.log.gz 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29001.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29905.log.gz 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.30236.log.gz 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29441.log 2024-09-16T03:24:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30237.log 2024-09-16T03:24:34.083 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29441.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29294.log.gz 2024-09-16T03:24:34.083 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25078.log 2024-09-16T03:24:34.083 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30237.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.30237.log.gz 2024-09-16T03:24:34.083 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28577.log 2024-09-16T03:24:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30434.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30434.log.gz 2024-09-16T03:24:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T03:24:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30616.log 2024-09-16T03:24:34.087 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.32692.log.gz 2024-09-16T03:24:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28709.log 2024-09-16T03:24:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32201.log 2024-09-16T03:24:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30616.log: /var/log/ceph/ceph-client.admin.28709.log: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32121.log 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30616.log.gz 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.28709.log.gz 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32201.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32201.log.gz 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29368.log 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33154.log 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33824.log 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32121.log.gz 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33154.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29368.log.gz 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33154.log.gz 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32744.log 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31616.log 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32482.log 2024-09-16T03:24:34.088 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33824.log.gz 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31616.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32744.log.gz 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.31616.log.gz 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30512.log 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30072.log 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32147.log 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32482.log.gz 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30072.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30512.log.gz 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30072.log.gz 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31128.log 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31942.log 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28552.log 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32147.log.gz 2024-09-16T03:24:34.089 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31942.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31128.log.gz 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.31942.log.gz 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31435.log 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28538.log 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32357.log 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28552.log.gz 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28538.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31435.log.gz 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28538.log.gz 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31682.log 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29712.log 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31682.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32357.log.gz 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29712.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31682.log.gz 2024-09-16T03:24:34.090 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.29712.log.gz 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32253.log 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30816.log 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30098.log 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30816.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32253.log.gz 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30816.log.gz 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30124.log 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33512.log 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30124.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30098.log.gz 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.30124.log.gz 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33330.log 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29264.log 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33330.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33512.log.gz 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33330.log.gz 2024-09-16T03:24:34.091 INFO:teuthology.orchestra.run.smithi045.stderr: 90.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31487.log 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32430.log 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32891.log 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29264.log.gz 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32430.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.31487.log.gz 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32430.log.gz 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31708.log 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32891.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21382.log 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33668.log 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32891.log.gz 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.tmp-client.admin.21382.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21382.log.gz 2024-09-16T03:24:34.092 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31708.log.gz 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29994.log 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32020.log 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33668.log.gz 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29994.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29994.log.gz 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30946.log 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32826.log 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30946.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32020.log.gz 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30946.log.gz 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28553.log 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28706.log 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28553.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.32826.log.gz 2024-09-16T03:24:34.093 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28553.log.gz 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29738.log 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28689.log 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29738.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28706.log.gz 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.29738.log.gz 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33564.log 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32796.log 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33564.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.28689.log.gz 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33564.log.gz 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33023.log 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29634.log 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33023.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32796.log.gz 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33023.log.gz 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31331.log 2024-09-16T03:24:34.094 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33850.log 2024-09-16T03:24:34.095 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31331.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29634.log.gz 2024-09-16T03:24:34.095 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31331.log.gz 2024-09-16T03:24:34.095 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31253.log 2024-09-16T03:24:34.095 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33486.log 2024-09-16T03:24:34.095 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33850.log: /var/log/ceph/ceph-client.admin.31253.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.33850.log.gz 2024-09-16T03:24:34.095 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33206.log 2024-09-16T03:24:34.095 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31253.log.gz 2024-09-16T03:24:34.098 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.25078.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.25078.log.gz 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25054.log 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28577.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.29441.log.gz 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28758.log 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.25054.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.25054.log.gz 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29851.log 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28758.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28577.log.gz 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28644.log 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29851.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28758.log.gz 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29133.log 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1%/var/log/ceph/ceph-client.admin.28644.log: -- replaced with /var/log/ceph/ceph-client.admin.29851.log.gz 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28898.log 2024-09-16T03:24:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29133.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29133.log.gz 2024-09-16T03:24:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T03:24:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28898.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.28898.log.gz 2024-09-16T03:24:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29199.log 2024-09-16T03:24:34.111 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mgr.x.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28506.log.gz 2024-09-16T03:24:34.111 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T03:24:34.113 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33486.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33486.log.gz 2024-09-16T03:24:34.113 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29056.log 2024-09-16T03:24:34.113 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30538.log 2024-09-16T03:24:34.113 INFO:teuthology.orchestra.run.smithi045.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T03:24:34.113 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33206.log: /var/log/ceph/ceph-client.admin.29056.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33206.log.gz 2024-09-16T03:24:34.113 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29056.log.gz 2024-09-16T03:24:34.113 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30486.log 2024-09-16T03:24:34.113 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30668.log 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30538.log: /var/log/ceph/ceph-client.admin.30486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29004.log 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30668.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30486.log.gz 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30538.log.gz 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30668.log.gz 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32331.log 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31734.log 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29004.log: /var/log/ceph/ceph-client.admin.32331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29816.log 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31734.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29004.log.gz 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32331.log.gz 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31916.log 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31734.log.gz 2024-09-16T03:24:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32383.log 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29816.log: /var/log/ceph/ceph-client.admin.31916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31154.log 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32383.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.31916.log.gz 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29816.log.gz 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32383.log.gz 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29968.log 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31357.log 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31154.log: /var/log/ceph/ceph-client.admin.29968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30460.log 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31357.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31154.log.gz 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29968.log.gz 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30642.log 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31357.log.gz 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32588.log 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30460.log: /var/log/ceph/ceph-client.admin.30642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33180.log 2024-09-16T03:24:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32588.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30460.log.gz 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30642.log.gz 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30150.log 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32588.log.gz 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29186.log 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33180.log: /var/log/ceph/ceph-client.admin.30150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32279.log 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29186.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33180.log.gz 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.30150.log.gz 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30020.log 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29186.log.gz 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29660.log 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.32279.log: /var/log/ceph/ceph-client.admin.30020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33720.log 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.29660.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30020.log.gz 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32279.log.gz 2024-09-16T03:24:34.116 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29660.log.gz 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30206.log 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31838.log 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.30206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28303.log 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.31838.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.33720.log.gz 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30206.log.gz 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28548.log 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31838.log.gz 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33238.log 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28303.log: /var/log/ceph/ceph-client.admin.28548.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28303.log.gz 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28548.log.gz 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32173.log 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28507.log 2024-09-16T03:24:34.117 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33238.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.32173.log: /var/log/ceph/ceph-client.admin.33101.log 2024-09-16T03:24:34.118 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.28507.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.33238.log.gz 2024-09-16T03:24:34.118 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32173.log.gz 2024-09-16T03:24:34.118 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31102.log 2024-09-16T03:24:34.118 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28507.log.gz 2024-09-16T03:24:34.118 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30046.log 2024-09-16T03:24:34.118 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.33101.log: /var/log/ceph/ceph-client.admin.31102.log: /var/log/ceph/ceph-client.admin.30046.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.33101.log.gz 2024-09-16T03:24:34.118 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31102.log.gz 2024-09-16T03:24:34.118 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30046.log.gz 2024-09-16T03:24:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29199.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.29541.log.gz 2024-09-16T03:24:34.119 INFO:teuthology.orchestra.run.smithi046.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.29199.log.gz 2024-09-16T03:24:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29522.log 2024-09-16T03:24:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28940.log 2024-09-16T03:24:34.123 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29522.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.29522.log.gz 2024-09-16T03:24:34.124 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29475.log 2024-09-16T03:24:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28940.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.29001.log.gz 2024-09-16T03:24:34.126 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T03:24:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30235.log 2024-09-16T03:24:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28498.log 2024-09-16T03:24:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30235.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29475.log.gz 2024-09-16T03:24:34.130 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.30235.log.gz 2024-09-16T03:24:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28787.log 2024-09-16T03:24:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29339.log 2024-09-16T03:24:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28498.log: /var/log/ceph/ceph-client.admin.28787.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.28940.log.gz 2024-09-16T03:24:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29062.log 2024-09-16T03:24:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29339.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28787.log.gz 2024-09-16T03:24:34.133 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29339.log.gz 2024-09-16T03:24:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29080.log 2024-09-16T03:24:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29079.log 2024-09-16T03:24:34.134 INFO:teuthology.orchestra.run.smithi046.stderr: 87.0%/var/log/ceph/ceph-client.admin.29080.log: -- replaced with /var/log/ceph/ceph-client.admin.29062.log.gz 2024-09-16T03:24:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29798.log 2024-09-16T03:24:34.136 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29079.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29080.log.gz 2024-09-16T03:24:34.137 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T03:24:34.138 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29798.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.29079.log.gz 2024-09-16T03:24:34.139 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29367.log 2024-09-16T03:24:34.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.audit.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29798.log.gz 2024-09-16T03:24:34.141 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28706.log 2024-09-16T03:24:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29367.log: 92.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T03:24:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29800.log 2024-09-16T03:24:34.155 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28706.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28706.log.gz 2024-09-16T03:24:34.156 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29904.log 2024-09-16T03:24:34.158 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29800.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28498.log.gz 2024-09-16T03:24:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29396.log 2024-09-16T03:24:34.159 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29800.log.gz/var/log/ceph/ceph-client.admin.29904.log: 2024-09-16T03:24:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29906.log 2024-09-16T03:24:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29396.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29904.log.gz 2024-09-16T03:24:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28536.log 2024-09-16T03:24:34.165 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29906.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.29906.log.gz 2024-09-16T03:24:34.166 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T03:24:34.171 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28536.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.29367.log.gz 2024-09-16T03:24:34.178 INFO:teuthology.orchestra.run.smithi046.stderr: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.28644.log.gz 2024-09-16T03:24:34.180 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28957.log 2024-09-16T03:24:34.181 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29527.log 2024-09-16T03:24:34.184 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28957.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.29396.log.gz 2024-09-16T03:24:34.189 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28973.log 2024-09-16T03:24:34.189 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28957.log.gz 2024-09-16T03:24:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29527.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29527.log.gz 2024-09-16T03:24:34.191 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T03:24:34.191 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28522.log 2024-09-16T03:24:34.193 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.4.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.28973.log.gz 2024-09-16T03:24:34.194 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29852.log 2024-09-16T03:24:34.208 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29853.log 2024-09-16T03:24:34.211 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29852.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29852.log.gz 2024-09-16T03:24:34.211 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29903.log 2024-09-16T03:24:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29853.log: 91.2% 93.8% -- replaced with /var/log/ceph/ceph-client.admin.29853.log.gz 2024-09-16T03:24:34.216 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28522.log.gz 2024-09-16T03:24:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28678.log 2024-09-16T03:24:34.220 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29903.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29903.log.gz 2024-09-16T03:24:34.220 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28530.log 2024-09-16T03:24:34.226 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28678.log: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.28536.log.gz 2024-09-16T03:24:34.238 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.28530.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28530.log.gz 2024-09-16T03:24:34.486 INFO:teuthology.orchestra.run.smithi046.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T03:24:34.741 INFO:teuthology.orchestra.run.smithi046.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.28678.log.gz 2024-09-16T03:24:34.902 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T03:24:35.208 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28628.log.gz 2024-09-16T03:24:35.446 INFO:teuthology.orchestra.run.smithi045.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T03:24:35.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-16T03:24:35.872 INFO:teuthology.orchestra.run.smithi046.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T03:24:41.432 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T03:24:41.467 INFO:teuthology.orchestra.run.smithi046.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T03:24:41.634 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T03:24:41.635 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T03:24:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m7.619s 2024-09-16T03:24:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m25.797s 2024-09-16T03:24:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m0.845s 2024-09-16T03:24:41.803 INFO:teuthology.orchestra.run.smithi045.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T03:24:41.820 INFO:teuthology.orchestra.run.smithi045.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T03:24:42.072 INFO:teuthology.orchestra.run.smithi045.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T03:24:42.074 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-16T03:24:42.074 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m8.097s 2024-09-16T03:24:42.074 INFO:teuthology.orchestra.run.smithi045.stderr:user 0m25.406s 2024-09-16T03:24:42.074 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m0.857s 2024-09-16T03:24:42.075 INFO:tasks.ceph:Archiving logs... 2024-09-16T03:24:42.076 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823/remote/smithi045/log 2024-09-16T03:24:42.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:24:45.545 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823/remote/smithi046/log 2024-09-16T03:24:45.546 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:24:48.789 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T03:24:48.798 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T03:24:48.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:24:48.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:24:48.884 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T03:24:48.892 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T03:24:48.892 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:24:48.895 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:24:48.906 INFO:teuthology.orchestra.run.smithi045.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:24:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================== 2024-09-16T03:24:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:+hv01.front.sepi 67.205.162.81 3 u 31 64 377 0.077 -2.623 0.235 2024-09-16T03:24:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:+hv02.front.sepi 63.231.80.2 3 u 38 64 377 0.107 -4.671 0.536 2024-09-16T03:24:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:*hv03.front.sepi 74.6.168.72 3 u 36 64 377 0.098 -3.528 0.294 2024-09-16T03:24:48.907 INFO:teuthology.orchestra.run.smithi045.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-16T03:24:48.908 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:24:48.908 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2024-09-16T03:24:48.908 INFO:teuthology.orchestra.run.smithi046.stdout:+hv01.front.sepi 67.205.162.81 3 u 35 128 377 0.079 -0.373 0.223 2024-09-16T03:24:48.908 INFO:teuthology.orchestra.run.smithi046.stdout:+hv02.front.sepi 63.231.80.2 3 u 99 128 377 0.106 -1.711 0.341 2024-09-16T03:24:48.909 INFO:teuthology.orchestra.run.smithi046.stdout:*hv03.front.sepi 74.6.168.72 3 u 26 128 377 0.087 -1.588 0.172 2024-09-16T03:24:48.909 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-16T03:24:48.909 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T03:24:48.920 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T03:24:48.921 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T03:24:49.005 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T03:24:49.092 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T03:24:49.191 INFO:teuthology.task.internal:Duration was 1812.240710 seconds 2024-09-16T03:24:49.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T03:24:49.246 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T03:24:49.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:24:49.249 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:24:49.289 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T03:24:49.289 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-09-16T03:24:49.289 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:24:49.339 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T03:24:49.340 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:24:49.350 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T03:24:49.350 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:24:49.383 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:24:49.418 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T03:24:49.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:24:49.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:24:49.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T03:24:49.532 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T03:24:49.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:24:49.546 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:24:49.561 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T03:24:49.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:24:49.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:24:49.600 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-09-16T03:24:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-09-16T03:24:49.634 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:24:49.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:24:49.670 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:24:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:24:49.681 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T03:24:49.691 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T03:24:49.692 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823/remote/smithi045 2024-09-16T03:24:49.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:24:49.737 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905823/remote/smithi046 2024-09-16T03:24:49.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:24:49.776 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T03:24:49.777 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:24:49.779 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:24:49.805 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T03:24:49.819 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T03:24:49.819 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T03:24:49.831 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T03:24:49.831 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:24:49.834 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:24:49.838 INFO:teuthology.orchestra.run.smithi045.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:24 /home/ubuntu/cephtest 2024-09-16T03:24:49.853 INFO:teuthology.orchestra.run.smithi046.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:24 /home/ubuntu/cephtest 2024-09-16T03:24:49.855 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T03:24:49.865 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T03:24:49.897 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} duration: 1812.240710258484 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T03:24:49.897 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:24:49.986 INFO:teuthology.run:pass